Charlie’s path to dEAth

Lucene is a great tool for retrieving fragmented information even including the fragmenting process (the Analyzer). So there was an intuitional way to retrieving molecules from a cheminformation database with the Lucene engine. ChemSink’s about (http://www.chemsink.com/about/) also mentions that "The chemical search uses Open Babel, Lucene, and MySQL ". Recently I have worked it out and have deployed on production services.

How it works

The concepts is connected as blow and the implementation is not hard. I work on my cheminformation platforms based on .NET and use Lucene.NET.

1. A Molecule is a File in Lucene.
2. Fingerprint is a Field. That means other information or even another suite of fingerprint can be stored for being queried.
3. A Bit in the fingerprint is a Term.
4. Substructure search queries Lucene for all the set bit in fingerprint of the query molecule are required in the result molecules.
5. Similarity search finds the most relevant molecules.

Similarity and Lucene

The most popular similarity algorithm being used is the Tanimato as written as

As my understanding, this coefficient is so widely used most because it’s simple and running fast for some time-critical cases such as online searching.

Lucene employs Cosin-Similarity with Vector Space Model (VSM) of Information Retrieval.

http://lucene.apache.org/java/3_0_0/api/all/org/apache/lucene/search/Similarity.html

and norm(t,d) part includes three parts to be multiplied, the lengthNorm part stands for what is effected by the length of the document (total fingerprint bits count of a molecule).

The default Similarity implementation inside Lucene is ready to be used for similar molecules retrieving.

More about the lengthNorm

In my researching phase I have found that the scores of several different target molecules are the same in value. It’s found that the lengthNorms are all the same for this several molecule while they have various number of fingerprint bits been set. The lengthNorm is not calculated while the searching phase but pre-calculated and stored at the indexing phase. Finally I have found this sentence inside the Lucene documents.

"However the resulted norm value is encoded as a single byte before being stored… comes with the price of precision loss"

So my molecules are treated as the same length documents when being queried.

The formula to calculate lengthNorm is 1.0 / Math.Sqrt(bits_set)

Fortunately, the DefaultSimilarity class could be overrided including the lengthNorm function. According to Duan Lian’s diagram of distribution of number of fingerprint bit set, there exists a function to mapping this distribution to a more flat one and been using to calculate lengthNorm and take full advantage of the precision-limited value.

 Here is the distribution of fingerprint darkness of my database of 80000 commercial compounds. 

Filed by charlie at 11:36 pm under Lucene, chemoinformatics
6 Comments

JSDraw是Tony Yuan开发的基于javascript的在线化学结构编辑器，也能作为化学结构式的显示工具。效果很棒。

我之前所了解的类似的软件是Chemhack jsMolEditor and Chemdoodle。

JSDraw的特点是

• 结构图片显示的质量高。
• 输入的方便。我用过的最方便的在线结构输入工具，是Chemwriter。而JSDraw的输入方式和Chemwriter几乎一样，尤其是直接用键盘输入N,S,O,Cl等非C原子的功能很有用。
• 有“部分选择”的功能，并支持Visio里一样的用ctrl键拖拽后复制的效果。
• 支持反映路线输入，并支持RXN导出。而且对反应路线输入，提供了自动整理格式的功能。
• 调用上很有特点。网页设计者不需要单独写js代码，而是直接对需要表达为结构式显示、编辑的区域，就是div元素，赋予特定的属性即可。
• 能够很完美支持IE。这点jsMolEditor还不行。

Tony现在在Novartis工作。提到这个公司，自然会想到一个人，就是Peter Ertl，大名鼎鼎的JME就是他多年前开发出来并广泛使用起来的。可见这个公司文化的开放和对行业多角度的贡献。

不过JSDraw并不是开源的，对商业应用也不免费。这一点Duan Lian的jsMolEditor就开放得多(LGPL)。

JSDraw目前还没有SMILES导出的功能，但是这在Tony的计划之中。如果我写的Javascript SMILES writer能够起到一些帮助，将是我很大的荣幸。

Filed by charlie at 11:23 pm under chemoinformatics
1 Comment

Indigo is recently introduced as open-source chemoinformatics toolkit <1>, <2>.

Dingo is a molecule and reaction rendering library included in Indigo and with .NET C# wrapper to make it possible for me to build a WPF app with it. Dingo and WPF make things so simple.

Amazing feeling to stand on the giant’s shoulder. Not too much code pasted here

Compound class to transform SMILES to Bitmap objects with Dingo

    public class Compound
    {
        public String smi{ get; set;}
        public Compound(String smiles)
        {
            smi = smiles;
        }
        public BitmapSource bmp {
            get{
                return Smi2Bitmap(smi, 160, 160);
            }
        }
        public static BitmapSource loadBitmap(System.Drawing.Bitmap source)
        {
            return System.Windows.Interop.Imaging.CreateBitmapSourceFromHBitmap(source.GetHbitmap(), IntPtr.Zero, Int32Rect.Empty,
                System.Windows.Media.Imaging.BitmapSizeOptions.FromEmptyOptions());
        }
        public static BitmapSource Smi2Bitmap(String smi, int Width, int Height)
        {
            Dingo dg = new Dingo();
            dg.loadMolecule(smi);
            dg.setBackgroundColor(System.Drawing.Color.White);
            System.Drawing.Bitmap bmp = dg.renderToBitmap(Width, Height);
            return loadBitmap(bmp);
        }
    }

SMILES file reader

    public class SMIReader: System.Collections.IEnumerable
    {
        private StreamReader sr;
        public SMIReader(String path)
        {
            sr = new StreamReader(path);
        }
        public IEnumerator GetEnumerator()
        {
            String line;
            while ((line = sr.ReadLine()) != null)
            {
                yield return new Compound(line.Split('\t')[0].Split(' ')[0]);
            }
        }
    }

XAML code of the listbox

Notes: To show as many structure pictures blocks on screen the WrapPanel is used as ItemsPanel. This depresses performance for large file that all SMILES are read in but lazy loaded in the default StackPanel mode.

            <ListBox Name="list1" Height="539" Width="790"
                     ItemTemplate="{StaticResource MolListboxTemplate}"
                     ScrollViewer.HorizontalScrollBarVisibility="Disabled"
                     IsSynchronizedWithCurrentItem ="True">
                <ListBox.ItemsPanel>
                    <ItemsPanelTemplate>
                        <WrapPanel/>
                    </ItemsPanelTemplate>
                </ListBox.ItemsPanel>
            </ListBox>

Item template of the listbox to support structure images displaying

    <Window.Resources>
        <DataTemplate x:Key="MolListboxTemplate">
            <WrapPanel Margin="3">
                <StackPanel>
                    <Image Source="{Binding bmp}" Height="160" Width="160"></Image>
                    <TextBox Text="{Binding smi}" Width="160"></TextBox>
                </StackPanel>
            </WrapPanel>
        </DataTemplate>
    </Window.Resources>

Button event to trigger databinding

        private void button1_Click(object sender, RoutedEventArgs e)
        {
            System.Windows.Forms.OpenFileDialog ofd = new System.Windows.Forms.OpenFileDialog();
            ofd.Filter = "SMILES file(*.smi)|*.smi";
            if (ofd.ShowDialog() == System.Windows.Forms.DialogResult.OK)
            {
                String path = ofd.FileName;
                textBlock1.Text = path;
                list1.ItemsSource = new nchem.SMIReader(path);
            }
        }

Done.

Filed by charlie at 11:40 pm under Coding, chemoinformatics
1 Comment

Rich在用ChemWriter的License作奖品请大家写一些结构输入软件的重要功能特性，进行竞赛。我不如就写点中文的吧。仅限于在线编辑的软件 ( online editor only )。

文件导入 (file importing)

没有谁的研究工作是在某个在线编辑器上进行的，更多的是文件形式存在的结构文件。比如Chemdraw的CDX文件。而重画一遍结构式，绝对是痛苦的事情。没有哪个研发助理和采购助理乐意干这件事。所以需要导入功能。导入功能在另一个侧面可以弥补功能上的不足，因为结构都已经在其他商业软件上做好了。

在线的结构文件导入，基于ActiveX的Chemdraw功能最全；硕大的JChemPaint可以支持InChI/SMILE等输入；Chemwriter可以导入mol文件；如果网页开发人员不多写一些代码，JME只能干画。

大小 (plugin size)

既然是在线使用，size就重要。比如JChemPaint有 382kB；Chemwriter 111kB；JME 40kB。

输入方便 (easy to input)

这一点Chemwriter可居榜首。鼠标在原子、键悬停时有显示；反复点击某个键更改键类型；直接在键盘上输入元素名称等等实用功能都做到了。

而且也比较美观。

所以综合来说，我很想用用Chemwriter，所以就写上面这些希望有点运气。

Filed by charlie at 10:20 pm under chemoinformatics, 不以物喜
No Comments

Yes, it is named by me as jsBabel. There’s too little functionality.

It is

• Written by Javascript.
• Implementation of span-tree generation and serialization.
• Modeling chemical compound in js.
• To help enhance the js molecule editor Chemhack jsmoleditor and Chemdoodle. They are excellent ideas but have no SMILES output function.
• In beta and many known and unknown errors.
• Going to be opensource if one or more folks is interested in it.
• Free to be downloaded, modified, re-published now.

A nice week end.

Filed by charlie at 4:21 pm under chemoinformatics, webdev
1 Comment

ChemDeposit是在Google Appengine (java) 上运行的化合物信息生成、存储、检索的服务。

在6月15号，就已经发布上去能工作的版本了，不过不能支持子结构检索，只能用指纹散列（fingerprint）做“近似子结构检索）。

为了调整CDK的版本，增加子结构检索(SMARTS Searching)，竟然用了一周的时间，遇到好多小问题。最难办的就是在客户机的SDK上运行OK，上传上去就不行的问题。最最难办的就是数据结构修改后，原有的数据成为非法数据删不掉的情况。

相关的问题和解决的过程在这里。

Jun 20, 2009

Version 3.

• Using CDK jar version 1.2.3
• Substructure Searching supported. That means the CDK SMARTS searching function works.
• Similar Structure Searching (Tanimoto) supported.
• Has a domain name www.chemdeposit.com

Filed by charlie at 10:56 pm under ChemDeposit, chemoinformatics
No Comments

Google App Engine has supported java for a while. I don’t know if others have tried to hosting CDK jar on the App Engine. Last weekend I tried to build an application or App Engine based on CDK and fortunately it seems most parts works.

Filed by charlie at 3:38 pm under ChemDeposit, Tools, chemoinformatics
1 Comment

Windows环境下，常用的SDF文件 阅读和处理的免费（或试用版）软件。

 

http://www.enso-software.com/WebSite2005/Default.aspx

http://www.hyleos.net/?s=applications&p=ChemFileBrowser

http://www.bioclipse.net/

基于eclipse。

Filed by charlie at 6:14 am under Tools, chemoinformatics
No Comments

A simple web interface is provided on SQLMOL now. It is writen in ASP.NET and jQuery is used from Google’s code hosting. The molecule structure images are generated by the Daylight‘s SMI2GIF service. SMI2GIF is so excellent that it can not only transform your SMILES to structure image but also highlight the substructure you have queried.

As SQLMOL is currently only implemented on SQL Server, you should have a MS Windows enviorenment to run it. Then I recommend the free tool Visual Web Developer to handle the source file and run the code to test.

SQLMOL is so simple and light weight substructure search system that can deployed on a common PC in half-an-hour. Just try it.

Filed by charlie at 7:34 pm under English, SQLMOL, chemoinformatics
No Comments

在后台用工具将结构式信息都转换成图片格式（PNG, JPEG等），再将图片上传到网站服务器，用普通的图片显示方式显示出来。需要注意的就是一般的产品目录中，产品数量较大，是用数据库生成的列表，批量生成的图片，就需要用产品编号、CAS编号等相对固定的编码对应起来。

后台生成图片的工具很多，大部分化学信息学软件都可以做得到。但是为了方便批量操作，尽量选择有编程接口的软件，至少也要能够通过命令行调用的软件。

使用在线结构图生成服务

可以利用的服务，我在之前的这篇在线化学结构式图片生成服务中详细介绍过。前提基本都是要有化合物的SMILES编码信息。这样做的好处是，不用自己生成图片，不用占用自己网站的空间，甚至不用占用自己网站的带宽。同样潜在的问题是要依赖第三方服务的准确性、可靠性和稳定性。

使用浏览器插件

使用浏览器插件恐怕是最常见的解决方案。一般来说，这样的插件都需要用结构式的mol格式作为输入，mol格式可以从SDFile的连接表部分导出。关于mol 和SDFile格式请参考 Chemical table files。目前也只有使用插件，能够实现结构式的输入。

最常见的网页插件有

JME, http://www.molinspiration.com/jme/，java applet

我所见过的90%以上的在web中显示和输入结构式的浏览器插件，都是用JME来实现的。JME的作者没有声称过这个软件是免费的，非商业用户需要向他发一个邮件来索要。而实际上既然是java applet，直接下载下来用也很容易。而事实上大多数的人都是这么做的。

这个软件估计至少有10年历史了，作者是Peter Ertl。

Jmol, http://jmol.sourceforge.net/, java applet

没有用过。开源的项目，支持的格式非常多，不过还是不支持SMILES/InChI。

Chemdraw plugin, http://www.cambridgesoft.com/software/details/?ds=2&dsv=93, activex

不是免费的，在工作中用到过。功能非常强大，结构式编辑功能与Chemdraw软件基本相当，远远超出其他插件。

Chemwriter, http://metamolecular.com/chemwriter/, java applet

Chemwrite应用并不广泛，功能也不强大，只能说图片显示的质量要比JME好不少，进行了边缘平滑的处理。之所以要提到，是因为这个软件的作者是 Depth-first 这个化学信息学博客的作者Rich Apodaca，我在这里学到了很多有用的知识。

JChemPaint, http://apps.sourceforge.net/mediawiki/cdk/index.php?title=JChemPaint, java applet

基于CDK项目的一个衍生产品。

基于Javascript和AJAX的方案

不论java applet还是activex，都对客户端有所要求。java applet要求客户访问网站的时候，要安装java运行时环境；activex默认在浏览器中不能运行，要客户判断安装并执行，因为有臭名昭著的安全问题。而基于Javascript和AJAX的方案，能够解决这些问题，也符合Web2.0的趋势。

MX-GWT, http://chemhack.com/mx-gwt/, 纯javascript

这是国内的一位叫Duan Lian的同学的作品。是我知道的和见过的唯一的用纯粹的javascript实现的结构式显示和输入方案。这个软件实际上是把Rich Apodaca用java开发的一个轻量级的化学信息处理库，MX，通过Google的GWT编译成javascript，使之能在浏览器里运行。非常棒。

Molinspiration WebME, http://www.molinspiration.com/docu/webme/index.html, AJAX

是开发JME的公司Molinspiration最近开发的AJAX的方案，需要服务器的支持。如果愿意买这个软件的话，Molinspiration提供服务器端支持。对客户来说，仍然只要能支持js的浏览器就行了。

PubChem edit, http://pubchem.ncbi.nlm.nih.gov/edit/, AJAX

PubChem在线服务的结构式输入工具，可以拆出来用。而且PubChem也足够开放，提供了很详细的使用说明，包括如何把输入的结构传递到表单。在在线化学结构式图片生成服务提到过，这个编辑器背后隐藏着一个通过SMILES来显示结构式的接口。

Filed by charlie at 9:33 pm under Tools, chemoinformatics, webdev
No Comments