Archive

Archive for the ‘SW Dev’ Category

Bye bye Sun and Solaris :-(

September 7th, 2017 No comments

So sad… but it’s inevitable, Oracle killing Solaris and Sun.

Oracle Finally Killed Sun

Categories: Java, Tech Tags: , , ,

Fair use of web content

August 11th, 2017 1 comment

This news was buried among many other news, but I felt that it deserves more people knowing about it.

It is about “fair use” of publicly available web content. What is “fair use” and when can content be restricted.

The original article is here.

A small company called hiQ is locked in a high-stakes battle over Web scraping with LinkedIn. It’s a fight that could determine whether an anti-hacking law can be used to curtail the use of scraping tools across the Web.

HiQ scrapes data about thousands of employees from public LinkedIn profiles, then packages the data for sale to employers worried about their employees quitting. LinkedIn, which was acquired by Microsoft last year, sent hiQ a cease-and-desist letter warning that this scraping violated the Computer Fraud and Abuse Act, the controversial 1986 law that makes computer hacking a crime. HiQ sued, asking courts to rule that its activities did not, in fact, violate the CFAA.

James Grimmelmann, a professor at Cornell Law School, told Ars that the stakes here go well beyond the fate of one little-known company.

I will leave it up to you to read and make up your own opinion about it.

Categories: Facts, news, Tech, web Tags: , , , ,

Using ldap and kerberos with ajaxplorer

November 16th, 2012 2 comments

12/18/12 Update: not all is peachy keen. Login and autocreate account works, but logout can be an issue. I need to clear the session cookie when someone logout. Have not gotten around to coding that yet.

After a bit of fiddling around, I finally got ajaxplorer working with (ldap) kerberos5 as the backend authentication/access.

We are using ldap for users directory and kerberos5 for password. It’s a little bit different than what I am used to.

Anyway, I needed to get ajaxplorer working on a large filer for users to be able to access — locally and remotely — essentially our private ‘dropbox’. But getting ajaxplorer working with kerberos was a bitch! At first, I tried using ldap, got that working…. except ldap does not have our password, that’s where kerberos comes in. I thought about writing my own plugin, but damn it, I don’t have time for this.

After lots of googling, experimenting, etc. I found mod_auth_pam, which uses pam for basic HTTP auth. And since we are already using pam_krb5 for logins on our boxes, it’s a perfect solution.

Here is the section in my bootstrap_plugins.php:

$PLUGINS = array(
        "CONF_DRIVER" => array(
                "NAME"          => "serial",
                "OPTIONS"       => array(
                        "REPOSITORIES_FILEPATH" => "AJXP_DATA_PATH/plugins/conf.serial/repo.ser",
                        "ROLES_FILEPATH"        => "AJXP_DATA_PATH/plugins/auth.serial/roles.ser",
                        "USERS_DIRPATH"         => "AJXP_DATA_PATH/plugins/auth.serial",
                        "FAST_CHECKS"           => false,
                        "CUSTOM_DATA"           => array(
                                        "email" => "Email",
                                        "country" => "Country"
                                )
                        )
        ),
        "AUTH_DRIVER" => array(
                "NAME"          => "basic_http",
                "OPTIONS"       => array(
                        "USERS_FILEPATH" => "AJXP_DATA_PATH/plugins/auth.pam/users.ser",
                        "AUTOCREATE_AJXPUSER"   => true,
                        "TRANSMIT_CLEAR_PASS"   => false
                )
        ),
        array(
                "NAME"          => "serial",
                "OPTIONS"       => array(
                        "LOGIN_REDIRECT"        => false,
                        "USERS_FILEPATH"        => "AJXP_DATA_PATH/plugins/auth.serial/users.ser",
                        "AUTOCREATE_AJXPUSER"   => false,
                        "FAST_CHECKS"           => false,
                        "TRANSMIT_CLEAR_PASS"   => false
                )
        ),
        "LOG_DRIVER" => array(
                "NAME" => "text",
                "OPTIONS" => array(
                        "LOG_PATH" => (defined("AJXP_FORCE_LOGPATH")?AJXP_FORCE_LOGPATH:"AJXP_INSTALL_PATH/data/logs/"),
                        "LOG_FILE_NAME" => 'log_' . date('m-d-y') . '.txt',
                        "LOG_CHMOD" => 0770
                )
        )
);

And the section in my /etc/httpd/conf.d/ajaxplorer.conf file:

   < Directory "/usr/share/ajaxplorer">
        Options FollowSymLinks
        AllowOverride Limit FileInfo
        Order allow,deny
        Allow from all
        AuthName "Ajaxplorer Access"
        AuthType Basic
        AuthPAM_Enabled on
        Require valid-user
  	php_value error_reporting 2
   < /Directory>

The trick is these two lines for the “basic_http” auth_driver:


"USERS_FILEPATH" => "AJXP_DATA_PATH/plugins/auth.pam/users.ser",
"AUTOCREATE_AJXPUSER" => true,

That then allow my users to login, and on first time, they auth via mod_auth_pam, and ajaxplorer create their account in “AJXP_DATA_PATH/plugins/auth.pam/users.ser”.

NOTE I have to manually create the directory plugins/auth.pam and create an empty users.ser file.

But after that, everything work perfectly.

Categories: SW Dev, Tech, web Tags: , ,

dynamic robots.txt file in Rails 3.x

April 27th, 2012 No comments

We have a need for dynamic handling of robots.txt file as we have different requirements for production, staging, dev, test, etc.

Google-fu shows various way to do this, some for Rails 2.x, some for Rails 3.x. Here is my version.

First is to edit config/routes.rb and add this line:


match '/robots.txt' => RobotsGenerator

Then add the following to app_root/lib/classes/robots_generator.rb.

NOTE: We have an old domain, foo.com, that redirects to our newfoo.com. We don’t want foo.com to get indexed, so I have special treatment for that in production

class RobotsGenerator
  # Use the config/robots.txt in production.
  # Disallow everything for all other environments.
  def self.call(env)
    req = ActionDispatch::Request.new(env)
    headers = {}
    body = if Rails.env.production?
      if req.host.downcase =~ /foo.com$/
        headers = { 'X-Robots-Tag' => "noindex,nofollow" }
        "User-agent: *\nDisallow: /"
      else
        File.read Rails.root.join('config', 'robots.txt')
      end
    else
        "User-agent: *\nDisallow: /"
    end

    [200, headers, [body]]
  rescue Errno::ENOENT
    [404, {}, "User-agent: *\nDisallow: /"]
  end
end

Finally, you want to move public/robots.txt to config/robots.txt.

I want to give credits to the people that inspired my version.

Categories: Rails, Ruby, Tech Tags: , ,

ActionMailer SSLError hostname was not match with the server certificate

March 15th, 2012 1 comment

So I am setting up a Continous Integration server using CruiseControl.rb and was getting these errors. I am on a RoR 3.1.x env and pointing to my local (same server) postfix for SMTP. I don’t need SSL.

OpenSSL::SSL::SSLError (hostname was not match with the server certificate):
/usr/lib/ruby/1.8/openssl/ssl.rb:123:in `post_connection_check’
/usr/lib/ruby/1.8/net/smtp.rb:582:in `tlsconnect’
/usr/lib/ruby/1.8/net/smtp.rb:562:in `do_start’
/usr/lib/ruby/1.8/net/smtp.rb:525:in `start’

http://scottiestech.info/2009/12/21/fixing-the-actionmailer-hostname-not-match-server-certificate-error/

Gave me a clue as to the problem. But adding this line

# Turn off auto TLS for e-mail
ActionMailer::Base.smtp_settings[:enable_starttls_auto] = false

to my config/environments/ci.rb does not work. I was still getting the same error. So poking around in ActionMailer gem source code gave me the last piece of clue I needed.

I also need to set this flag:

:openssl_verify_mode => false

Putting everything together,


ActionMailer::Base.smtp_settings[:openssl_verify_mode] = false
ActionMailer::Base.smtp_settings[:enable_starttls_auto] = false

is all I need. Turning off starttls is not needed, but I’d do it anyway because I am talking to my local SMTP server and don’t want the overhead.

Categories: Email, Rails, Ruby, SW Dev, Tech Tags: , ,

google-spreadsheet-ruby and (Nokogiri::XML::XPath::SyntaxError)

March 13th, 2012 No comments

I have a Ruby script that update our Google Doc spreadsheet as part of releases. The script has been working for years, then suddenly started failing with errors like this one:


/usr/local/lib/ruby/gems/1.9.1/gems/nokogiri-1.5.2/lib/nokogiri/xml/node.rb:159:in `evaluate': Invalid expression: .//xmlns:link[@xmlns:@rel = 'http://schemas.google.com/spreadsheets/2006#cellsfeed'] (Nokogiri::XML::XPath::SyntaxError)
from /usr/local/lib/ruby/gems/1.9.1/gems/nokogiri-1.5.2/lib/nokogiri/xml/node.rb:159:in `block in xpath'
from /usr/local/lib/ruby/gems/1.9.1/gems/nokogiri-1.5.2/lib/nokogiri/xml/node.rb:150:in `map'
from /usr/local/lib/ruby/gems/1.9.1/gems/nokogiri-1.5.2/lib/nokogiri/xml/node.rb:150:in `xpath'
from /usr/local/lib/ruby/gems/1.9.1/gems/nokogiri-1.5.2/lib/nokogiri/xml/node.rb:214:in `css'
from /usr/local/lib/ruby/gems/1.9.1/gems/google-spreadsheet-ruby-0.1.6/lib/google_spreadsheet.rb:648:in `block in worksheets'
from /usr/local/lib/ruby/gems/1.9.1/gems/nokogiri-1.5.2/lib/nokogiri/xml/node_set.rb:239:in `block in each'
from /usr/local/lib/ruby/gems/1.9.1/gems/nokogiri-1.5.2/lib/nokogiri/xml/node_set.rb:238:in `upto'
from /usr/local/lib/ruby/gems/1.9.1/gems/nokogiri-1.5.2/lib/nokogiri/xml/node_set.rb:238:in `each'
from /usr/local/lib/ruby/gems/1.9.1/gems/google-spreadsheet-ruby-0.1.6/lib/google_spreadsheet.rb:646:in `worksheets'
from /home/tin/bin/google-spreadsheet:125:in `

'

It took a bit of sleuthing to find out that Nokogiri version 1.5.2 broke google-spreadsheet-ruby. Downgrading to v 1.5.0 made it work again.

I am documenting it here for those of you running into similar problems.

Addtions. It turns out nokogiri and google-spreadsheet gems are written by same author, and he is aware of the bug, but not fixing it at this time. He recommends using nokogiri v1.5.0

https://github.com/gimite/google-spreadsheet-ruby/issues/31#issuecomment-4451659

Google-fu show a solution, since I can’t keep gems from updating on my systems. I just have to force my script to request a particular version.

http://stackoverflow.com/questions/2693862/how-do-i-require-a-specific-version-of-a-ruby-gem

And here is how I solved my problem.


gem 'nokogiri', '=1.5.0'

require 'optparse'
require 'ostruct'
require 'csv'
require 'rubygems'
require 'nokogiri'
require 'google_spreadsheet'

Categories: Ruby, SW Dev Tags: , , ,

Summary of Overcoming RoR Performance Challenges Meetup on Wed 2/29/12

March 1st, 2012 No comments

Overcoming RoR Performance Challenges Meetup

The talk was on best practices and some tips on looking for problems and how the panelists worked around them. There are no “magic” bullet like Ruby or RoR has 🙂

Essentials:

  1. Watch out when using ActiveRecord. It make it too easy to use DB. It make it too easy to use DB. One more time, it make it too easy to use DB.

    Essentially, ActiveRecord and DB is not always the right tool. Sometime using other tool could work better for a particular problem.

    Things mentioned:

    • Using Redis as a queueing system, to buffer writes, which later go to DB. (this is what Blitz, Bleacher Report use to increase their performance).
    • Use NoSQL (CouchBase, Mongo and Cassandara were mentioned as being used by panelists).
    • Cache results as much as possible. Don’t hit DB all the time.
    • Hand optimize queries might be needed. ActiveRecord is not the best at generating optimized DB calls.
  2. Cache as much as possible. Bleacher Reports put in caching layer everywhere, memcache, front end web cache, etc. They also have scripts that pre-warmed their cache (“goal is to never have users be the one who triggered a cache request”).

    Use the cache in newer RoR (3.2).

  3. Write code in ways that make it easy to update to latest Ruby and RoR.

    Ruby EE has flags to allow you to use more memory for internal cache. Sometime it make sense to test for and try different memory configuration there (based on 2 panelists’ experiences).RoR 3.2 has good Rack/Rails cache. Read the doc and use them.

  4. Background processes.
    • Use bg proc whenever possible.
    • Anytime you need to make calls to external website (external API), use a bg process, to not tie up your RoR web process.
    • Blitz put jobs into Redis queue, then bg server check Q for job, run it and put partial results back into Redis, Ajax call then check and format/display result to web client.
    • Bleacher Reports and Mixbooks also do similar things. They use Redis as a job queueing system, among other things (see 1 above).
  5. They all mention using other web server for production (not using webrick). The following were mentioned as being used by panelists.
    • Passenger
    • Thin
    • Unicorn
  6. Related to (ActiveRecord) above is the N+1 problem. Where you add 1 line of code and the DB calls increased manifold.
    • Advice essentially say to develop and use coding best practices and train developers to look out for them.
    • There is a possible test that can be use to automated looking out for N+1 issue.
    • Solving n+1 problem with special tests: http://en.oreilly.com/rails2009/public/schedule/detail/8615 Query testing – see PDF of slides page 75
    • Panelists all recommended RSpec for automated testing.
  7. Monitoring for issues and performance.
    • All panelists point to NewRelic as the tool they use all the time.
    • The host of the meeting Blitz also did a marketing spiel on their tool to use for performance testing (it look really good, and available as a plugin on Heroku). I am going to test it and see about using it for performance/load testing our site.
  8. For ease of scaling infrastructure, leverage AWS EC2, Heroku, Engine Yard and other cloud providers.
Categories: Cloud, EC2, Heroku, Rails, Ruby, SW Dev, Tech Tags: , , , , , , ,

Errors while installing unicorn gem on OS X

February 2nd, 2012 No comments

I was updating my bundle for a rails 3.1 app, when I run into the following errors.

Installing unicorn (4.2.0) with native extensions
Gem::Installer::ExtensionBuildError: ERROR: Failed to build gem native extension.

/opt/local/bin/ruby1.9 extconf.rb
checking for SIZEOF_OFF_T in ruby.h... *** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of
necessary libraries and/or headers. Check the mkmf.log file for more
details. You may need configuration options.

Provided configuration options:
--with-opt-dir
--with-opt-include
--without-opt-include=${opt-dir}/include
--with-opt-lib
--without-opt-lib=${opt-dir}/lib
--with-make-prog
--without-make-prog
--srcdir=.
--curdir
--ruby=/opt/local/bin/ruby1.9
/opt/local/lib/ruby1.9/1.9.1/mkmf.rb:381:in `try_do': The compiler failed to generate an executable file. (RuntimeError)
You have to install development tools first.
from /opt/local/lib/ruby1.9/1.9.1/mkmf.rb:491:in `block in try_compile'
from /opt/local/lib/ruby1.9/1.9.1/mkmf.rb:443:in `with_werror'
from /opt/local/lib/ruby1.9/1.9.1/mkmf.rb:491:in `try_compile'
from /opt/local/lib/ruby1.9/1.9.1/mkmf.rb:686:in `macro_defined?'
from /opt/local/lib/ruby1.9/1.9.1/mkmf.rb:822:in `block in have_macro'
from /opt/local/lib/ruby1.9/1.9.1/mkmf.rb:790:in `block in checking_for'
from /opt/local/lib/ruby1.9/1.9.1/mkmf.rb:284:in `block (2 levels) in postpone'
from /opt/local/lib/ruby1.9/1.9.1/mkmf.rb:254:in `open'
from /opt/local/lib/ruby1.9/1.9.1/mkmf.rb:284:in `block in postpone'
from /opt/local/lib/ruby1.9/1.9.1/mkmf.rb:254:in `open'
from /opt/local/lib/ruby1.9/1.9.1/mkmf.rb:280:in `postpone'
from /opt/local/lib/ruby1.9/1.9.1/mkmf.rb:789:in `checking_for'
from /opt/local/lib/ruby1.9/1.9.1/mkmf.rb:821:in `have_macro'
from extconf.rb:4:in `

'

Gem files will remain installed in /Users/tin/.bundler/tmp/34826/gems/unicorn-4.2.0 for inspection.
Results logged to /Users/tin/.bundler/tmp/34826/gems/unicorn-4.2.0/ext/unicorn_http/gem_make.out
An error occured while installing unicorn (4.2.0), and Bundler cannot continue.
Make sure that `gem install unicorn -v '4.2.0'` succeeds before bundling.

And the mkmf.log contain this, which was the clue I needed.

"/usr/bin/gcc-4.2 -o conftest -I/opt/local/include/ruby-1.9.1/x86_64-darwin10 -I/opt/local/include/ruby-1.9.1/ruby/backward -I/opt/local/include/ruby-1.9.1 -I. -I/opt/local/include -D_XOPEN_SOURCE -D_DARWIN_C_SOURCE -I/opt/local/include -pipe -O2 -arch x86_64 -fno-common conftest.c -L. -L/opt/local/lib -L/opt/local/lib -L. -L/opt/local/lib -arch x86_64 -L/usr/local/lib -lruby.1.9.1 -lpthread -ldl -lobjc "
i686-apple-darwin10-gcc-4.2.1: vfork: Operation timed out
checked program was:
/* begin */
1: #include "ruby.h"
2:
3: int main() {return 0;}
/* end */

Notice the vfork: Operation timed out line? It turns out that I was running out of system resources (VM) and gcc can’t fork. Ah, my poor 8GB i7 laptop…. 🙂

Fix was simple, I shutdown Firefox and Chrome w/their gadzillion windows and tabs opened. Then retried “bundle update” again, and voila!

Documenting it here in case other people run into similar problem.

Categories: OSX, Ruby, SW Dev, Tech Tags: , , , ,

recover / reset lost posgres password

May 30th, 2009 No comments

I am a long time MySQL user, so Postgres administration is unfamiliar to me.  I did used Postgres a long time ago… when I first started my web hosting company, but back then (early 90’s), Postgres was young and not suited for our needs.  Discovered MySQL and never looked back.

Yeah, yeah, stop with the religious war already.  I believe that using the right tool for the job is more important than anything else.

Back to the problem at hand.  As a new, incoming admin, there is a lot of things I have to pick up on-the-fly.  There is a lot of historical knowledge that I do not have at new place.  Such as Postgres, which *work* uses and now I have to admin.  It seem that no one know the postgres user password, or don’t want to share ;->

I really don’t care for politics, just want to do my work!  So things break and I need access to system tables to fix it.  Can’t login to Postgres as postgres, yes, I have root and can su postgres, but still can not login via psql.

E.g. psql -h localhost -U postgress -d template1  ask me for password and I don’t know postgres user password

Searching (googling) found a lot of people asking for help, etc. but no specific way to solve the problem.  At least for a newbie Postgres admin like me.

Finally, someone pointed to pg_hba.conf…. reading the manual, backward, forward, sideways, etc…. and finally, finally, figured it out!

Here it is to save time for others.

Edit pg_hba.conf (usually in /var/lib/psql/data directory, at least on RH/CentOS/Fedora).

Add this line at top, first ACL match wins, so don’t worry about the rest there.

host all postgres 127.0.0.1/32 trust

Then at command line, run pg_ctl reload

Now you can psql -h localhost -d template1 -U postgres and login without password.  Change the password to something you know.

alter user postgres with encrypted password ‘newpassword’;

Exit and comment out the line you added to pg_hba.conf, then reload config via

pg_ctl reload

Tada!  All fixed.

Categories: Postgresql, SW Dev, Tech Tags: ,

Monitoring Java programs

May 10th, 2009 2 comments

It’s harder than it look, or perhaps I am just making it harder than it really is.  Anyway, I have needs to monitory performance of java based applications.

There are actually two types (to me) of Java apps.  There is the standalone apps that you run on your workstation (Eclipse, I count embedded web applets in this category), and then there is the server based types such as Tomcat/JSP/J2EE/etc.

The category that I am most interested in is the server based apps.  I need to be able to look inside the JVM they are running in, and also the container (Tomcat/J2EE/etc.).  I am not an expert in this area, so at the moment, it’s a blackbox to me.

I’ve been searching around (yes, started with Googling :-)) and found lots of information all over the place.  I am going to try to gather them into one spot for my benefits, and hopefully save others some time.  As always, if you have corrections, additions, please feel free to send them to me.

List of Java performance monitoring tools.

  • this came from here
    1. jconsole comes with JDK 1.5 and above. It is a Java Monitoring and Management Console – JMX-compliant graphical tool for monitoring a Java virtual machine. It can monitor both local and remote JVMs.
    2. VisualVM is a visual tool that integrates several existing JDK software tools and lightweight memory and CPU profiling capabilities. This tool is designed for both production and development time use and further enhances the capability of monitoring and performance analysis for the Java SE platform.
    3. HeapAnalyzer allows the finding of a possible Java™ heap leak area through its heuristic search engine and analysis of the JavaTM heap dump in Java applications. It analyzes Java heap dumps by parsing the Java heap dump, creating directional graphs, transforming them into directional trees, and executing the heuristic search engine.
    4. PerfAnal is a GUI-based tool for analyzing the performance of applications on the Java 2 Platform. You can use PerfAnal to identify performance problems in your code and locate code that needs tuning.
    5. JAMon is a free, simple, high performance, thread safe, Java API that allows developers to easily monitor production applications.
    6. Eclipse Memory Analyzer is a fast and feature-rich Java heap analyzer that helps you find memory leaks and reduce memory consumption.
    7. GCViewer is a free open source tool to visualize data produced by the Java VM options -verbose:gc and -Xloggc:<file>. It also calculates garbage collection related performance metrics (throughput, accumulated pauses, longest pause, etc.).
    8. HPjmeter
      • Identify and diagnose performance problems in Java™ applications running on HP-UX
      • Monitor live Java™ applications and analyze profiling data
      • Capture profiling data with zero preparation when using JDK/JRE 5.0.04 or higher
      • Run the HPjmeter console on HP-UX, Linux, and Windows® systems
      • Improve garbage collection performance
    9. HPjconfig is a Java configuration tool for tuning your HP-UX 11i HP Integrity Itanium® and HP 9000 PA-RISC system kernel parameters to match the characteristics of your application. HPjconfig provides kernel parameter recommendations tailored to your HP-UX hardware platform. It offers save and restore functions for easy distribution of tailored recommendations across your customer base. When given specific Java and HP-UX versions, HPjconfig will determine if all of the latest HP-UX patches required for Java performance and functionality are installed on the system, and highlight any missing or superseded patches.
    10. Java Out-of-Box Tool is a stand-alone bundle that upon installation will install startup (RC) scripts, modify kernel parameters, rebuild the kernel, and reboot the system. During startup, the startup scripts will modify system tunables, thus providing better “Out of The Box” behavior for Java.
  • eclipse TPTP and Netbeans Profiler
  • YourKit
  • JProfiler
  • JIP
  • crap4j
  • JRockit

12/12/09 Updates

Yes, I will also consider commercial tools.  It’s been a few years since I develop Java full time.  My focus these days are on the Operation side – Network/System Administration, architectures, day to day operations.

Categories: Java, SW Dev, Tech Tags: , , ,