tag:blogger.com,1999:blog-26181356605185675892024-03-05T18:22:45.011-03:00*NULL Pointer AssignJosé Luis Campanellohttp://www.blogger.com/profile/00233954763631710324noreply@blogger.comBlogger17125tag:blogger.com,1999:blog-2618135660518567589.post-59001281916958418482013-12-31T18:59:00.002-03:002014-01-05T12:31:13.708-03:00Time flies!<div dir="ltr" style="text-align: left;" trbidi="on">
Another year... It seems like i've just blinked.<br />
<br />
I hope you have a very good new year.<br />
<br />
Have you make any donations this year?<br />
<br />
For my part, i continued my monthly donations to the WFP (held by United Nations, <a href="http://www.wfp.org/">www.wfp.org</a>) and to Nuru International (<a href="http://www.nuruinternational.org/">www.nuruinternational.org</a>). The first program is to bring relief, the second with the intention to really solve the problem. Let's hope.<br />
<br />
As for my end-of-year donations, this time i donated to:<br />
<br />
<br />
<div>
<ul style="text-align: left;">
<li>Wikimedia Foundation. Donation page in <a href="https://donate.wikimedia.org/w/index.php?title=Special:FundraiserLandingPage&country=AR&uselang=en&utm_medium=spontaneous&utm_source=fr-redir&utm_campaign=spontaneous" target="_blank">english</a> and <a href="https://donate.wikimedia.org/w/index.php?title=Special:FundraiserLandingPage&country=AR&uselang=es&utm_medium=sidebar&utm_source=donate&utm_campaign=C13_es.wikipedia.org" target="_blank">spanish</a>. They accept donations in ARS (Peso Argentino)<strike>, which is a good one to avoid the 35% extra charge for USD expenses</strike>. <b>Update</b>: the transaction proceeds actually in USD.</li>
<li>SETI Institute. Donation <a href="https://www.teamseti.org/donate" target="_blank">page</a>.</li>
<li>Unicef Argentina. I made the donation thru my bank.</li>
</ul>
</div>
<div>
For my dismay, i tried to donate to Cruz Roja Argentina (Argentina's Red Cross chapter). If you can believe it, the donation page requests you put your credit card information, but the page is not secure (plain old HTTP instead of HTTPS). This is a big WTF! I contacted them explaining the issue and requested a different means to donate. Lets see how this goes.<br />
<b>Update</b>: they have responded me, indicating that they do care about that and they use a secure payment processor (they page is still HTTP). I requested an account number to donate safely.</div>
<div>
<br /></div>
<div>
Well, i hope you have a happy new year, electric power if you live in Buenos Aires (another big WTF this year) and an open wallet to donate some money.</div>
<div>
<br /></div>
<br />
<div>
<div>
This is enough to be my seventeenth post.</div>
</div>
</div>
José Luis Campanellohttp://www.blogger.com/profile/00233954763631710324noreply@blogger.com1tag:blogger.com,1999:blog-2618135660518567589.post-29544547812033846062013-11-17T18:30:00.000-03:002013-11-17T18:31:02.116-03:00Problems and Obstacles<div dir="ltr" style="text-align: left;" trbidi="on">
I was recently talking with colleagues, discussing an issue and, unexpectedly, i asked: "is this issue a problem or an obstacle?"<br />
<br />
While i spoke the words, i remember thinking that i needed to devote some brain cycles to think about it. Hence this post.<br />
<br />
The first thing to understand is we usually use the terms interchangeably, but they mean different things (see Wiktionary's definition for <a href="http://en.wiktionary.org/wiki/problem#Noun" rel="nofollow" target="_blank">problem</a> and <a href="http://en.wiktionary.org/wiki/obstacle#Noun" rel="nofollow" target="_blank">obstacle</a>).<br />
<br />
A Problem is something active, that you interact with and that can even change you. An Obstacle is passive, something that you have to circumvent. Unnecessary.<br />
<br class="Apple-interchange-newline" />
Being a natural engineer of sorts, i like Problems. I don't very much like obstacles. But both require energy.<br />
<br />
Many startups become successes by solving Problems, but probably many more thrive by solving one type of Obstacle for their customers (the startup's problem is then to efficiently and scalably solve said obstacle).<br />
<br />
Normally, companies build Obstacles to enforce specific behaviours into their workers. Yet, the worst expression of Obstacle comes to be when companies turn this behaviour-enforcing Obstacles into cysts that plague work. Bureaucracies are just cysted piles of Obstacles.<br />
<br />
As a worker, you normally should scale up Problems to your boss and handle Obstacles yourself, but this is not always possible. Sometimes your boss is an Obstacle on it's own. Sometimes the company you work for loves to make Problems out of Obstacles, so that no employee can ever make any decision. Sometimes you are the Obstacle... or Problem.<br />
<br />
Yes. We all sometimes create Obstacles and Problems for ourselves or others. We usually do so because of our personalities, fears or bad habits. The worst self-delusional use of Obstacles is to hide Problems (circumventing something is a nice way of not examining that something). Another annoying personality trait is to turn Obstacles into Problems (aka: the resident renegade).<br />
<br />
For some reason that escapes me, Obstacles tend to rapidly become part of the Culture, while Problems tend to remain ignored. Yet, Problems usually hide a huge opportunity for economic, personal and/or social growth.<br />
<br />
Finally, a life with only Obstacles makes you feel worthless; a life with only Problems makes you feel overwhelmed.<br />
<br />
Learn to tell Problems from Obstacles. May your Obstacles be small and your Problems sparse. Fill the rest with satisfaction or happiness.<br />
<br />
This is enough to be my sixteenth post.</div>
José Luis Campanellohttp://www.blogger.com/profile/00233954763631710324noreply@blogger.com0tag:blogger.com,1999:blog-2618135660518567589.post-32004792162482626612013-02-27T15:58:00.001-03:002013-02-27T15:58:03.354-03:00Adiós Fibertel, y Fuck You!<div dir="ltr" style="text-align: left;" trbidi="on">
Si. Luego de 4 años y unos pocos días, estoy cambiando de proveedor. No más Fibertel. Ahora es Telecentro. Porqué? Bueno, lo que sigue explica el motivo, pero lo más importante viene después, porque creo que hay varias lecciones que aplican a los negocios y hay que tener siempre presente.<br />
<br />
<h3 style="text-align: left;">
Breve Historia</h3>
<div>
<br /></div>
<div>
Unos días antes del 2 de Octubre del 2012 (hace casi 5 meses), mi servicio de Internet murió. Las luces del modem iban y venían, pero la más importante (Cable) estaba apagada. Mi modem no lograba negociar con su contraparte. Llamo al 0-810-algo y me asignan una visita de técnico para el 2 de Octubre. Un fastidio. Sin internet? Puedo vivir sin gas (gracias delivery), sin TV (no hay gran cosa para ver?) pero no sin Internet. Uff. Bueno.</div>
<div>
<br /></div>
<div>
Orden de trabajo 6810763. Vino el técnico y revisó todo. Diagnóstico: el problema es que no llega suficiente señal a la caja distribuidora del edificio. Tienen que cambiar el cable. Uh, que? Y cuanto tarda? No, lo hacen dentro de las 72 horas. Y tengo que hacer algo? No, es un tema global del edificio así que la empresa se organiza para hacerlo. Ah, bueno.</div>
<div>
<br /></div>
<div>
Dos días después, el servicio se restauró. Bien!</div>
<div>
<br /></div>
<div>
Y viví feliz, hasta el 28 de Diciembre del 2012 (si, justo el día del inocente). Me lleva el diablo! (28-Dic-2012 fue viernes, con 31 y 1 lunes y martes, feriados). Sabía lo que significaba: 1 semana sin internet.</div>
<div>
<br /></div>
<div>
Llamé a servicio técnico. Me dieron visita para el 4 de Enero. No sabían si era algo específico, general de la zona o estaban trabajando en algún punto, así que podía arreglarse solo. El 2 de Enero, como para comenzar bien el año, la cosa se había arreglado. Cuando el servicio automático para confirmar las visitas llamó, la cancelé.</div>
<div>
<br /></div>
<div>
Cerca del 20 de Enero tuve otro corte (otro sábado), que duró hasta el domingo.</div>
<div>
<br /></div>
<div>
El último acto comenzó el 9 de Febrero. El servicio se fue nuevamente, otra vez con un fin de semana largo por delante (11 y 12 de carnaval).</div>
<div>
<br /></div>
<div>
Para servicio técnico digite 1. Visita para el 12-Feb, Orden de Trabajo 26849786. Y esta vez, pensé, aún si se arregla no voy a cancelar la visita. Me dije que algún problema debía haber, mientras comenzaba a sospechar que el arreglo de Octubre nunca había ocurrido.</div>
<div>
<br /></div>
<div>
Y así fue que vino el técnico y revisó todo. Diagnóstico: el problema es que no llega suficiente señal a la caja distribuidora del edificio. Tienen que cambiar el cable. Si, mismo diagnóstico que el de Octubre. Lo sabía!</div>
<div>
<br /></div>
<div>
Ya no tan feliz, pero diciéndome todo el mundo se equivoca, llamé al servicio técnico el miércoles 13 de Febrero. Orden de trabajo 35744811. Vienen el viernes 15 a reinstalar la subida entre 8 y 18 (bueno, es algo que normalmente hacen con los encargados, que están todo el día). Les aclaré que toquen en mi departamento ya que el encargado está de vacaciones, pero que yo tenía temporariamente las llaves para la terraza.</div>
<div>
<br /></div>
<div>
El jueves 14, como relojito, el sistema automático llamó para confirmar la visita. "Si confirma la visita presione 1". Claramente, presioné 1.</div>
<div>
<br /></div>
<div>
Viernes 15 de Febrero de 2013. El gran día! Me levanté 7:30 (inaudito para mis costumbres de búho). Y me puse a leer mientras esperaba. Sin Internet no se puede trabajar ni hay mucho que hacer. La caja boba se volvió virtual ;-)</div>
<div>
<br /></div>
<div>
Alrededor de 13:30 decidí salir 20 minutos a comprar algo de comida, así que, precavido el tipo, llamé para preguntar si tenían idea de a que hora iba a venir el técnico, ya que tenía que salir por unos minutos a hacer una compra.</div>
<div>
<br /></div>
<div>
No. La visita me figura como anulada. Anulada? Porqué anulada. Yo la confirmé ayer a la máquina que me llamó (se ve que sólo les importa su tiempo, no el de los clientes). Si, no sabría decirle porqué está anulada. No me figura el motivo. Pero le estoy tomando el reclamo y se tiene que resolver en 72 horas así que lo van a llamar por esto. El reclamo es el número 36624236.</div>
<div>
<br /></div>
<div>
Mientras transcurría la conversación, y ya sin tan buen espíritu, pensaba que una de tres cosas estaban pasando, estos tipos: 1) no son capaces de resolver el problema, 2) no quieren resolver el problema o 3) me están tomando por boludo.</div>
<div>
<br /></div>
<div>
<b>Resolución JLC-98902342</b>: 72 horas para resolver el problema, caso contrario, resolver personalmente el asunto (en criollo, Telecentro).</div>
<div>
<br /></div>
<div>
Así que esperé. Semana con un miércoles feriado, pero 4 días hábiles. Tiempo más que suficiente (96 hs > 72 hs). Pero no me llamaron...</div>
<div>
<br /></div>
<div>
Lunes 25 de Febrero. Sonó el despertador. Arriba. Al baño. A la cocina. A chequear el mail. No hay servicio.</div>
<div>
<br /></div>
<div>
<b>Ejecutando resolución JLC-98902342</b>:</div>
<div>
<ol style="text-align: left;">
<li>Se ha comunicado con el nuevo centro de atención al cliente de Telecentro.</li>
<li>Se ha comunicado con Cablevisión - Fibertel. Presione 1 si desea... Hola, si, llamo para dar la baja del servicio Fibertel. Lo comunico con el centro de atención especializado... Mirá, entiendo que tengas que decir esto y aquello, pero por favor no des más vueltas, ya pagué el costo de instalación de Telecentro así que no hay vuelta atrás, tomá la baja. ¿Está consciente de que va a tener una menor calidad de servicio? Uh, prefiero menor calidad de servicio con Telecentro a un servicio que no tengo con Fibertel. Bueno señor, le confirmo la baja con el número de trámite 38087865. Esto es a partir del primero de marzo, verdad? A partir del 31 de marzo, se le va a cobrar el mes de marzo. Yo pensé... y dije OK, con un cierto entusiasmo.</li>
</ol>
</div>
<div>
27 de Febrero, 9:40. Buenos días señor lo llamo del centro de gestiones especiales de Cablevisión. Quería preguntarle el motivo de la baja, blah blah. Si, ustedes saben el motivo. Revisá la historia de tickes y de como me dejaron de garpe y que estoy sin servicio. Gracias! Hasta luego! Y corté. La verdad, ni siquiera me preguntó si tenía posibilidad de hablar en ese momento. Si, porque obviamente yo estoy al servicio de Fibertel.</div>
<div>
<br /></div>
<blockquote class="tr_bq">
<b>Adiós Fibertel, y FUCK YOU!!!!</b> </blockquote>
<blockquote class="tr_bq">
Defensa del Consumidor, nos vemos pronto!</blockquote>
<div>
Defensa del Consumidor? Bueno si. Me refiero a ese OK con entusiasmo. Es el <i>asmo</i> equivocado, fue más bien sarcasmo. Así que no sólo me querés mojar la oreja, también querés escupirme en el ojo. Si claro, uno de estos días!</div>
<div>
<br /></div>
<div>
Hasta ese momento, todo el asunto era como un animal muerto en la ruta. Lo pasás y ya no pensás en el pobre bicho aplastado en el asfalto. No tiene mucho sentido ir por la vida buscando venganza por cada estupidez que ocurre...</div>
<div>
<br /></div>
<div>
Pero pagar por marzo de un servicio que estoy dando de baja porque básicamente no tengo servicio? <b>No fucking way!</b> Porque eso de pagar marzo es probablemente alguna cosa administrativa interna de Fibertel con su sistema de facturación.</div>
<div>
<br /></div>
<div>
Y ya que estamos, me van a devolver los días que no tuve servicio. Y me van a pagar las 5:30 horas que me dejaron de garpe el 15 de Febrero.</div>
<div>
<br /></div>
<div>
Y por cierto, el 11 de Febrero compré un modem 3G y le puse 60 mangos, porque no tenía servicio y tenía que laburar. Y el lunes 25 tuve que volver a cargar 50 mangos al bendito modem, para laburar. Eso también van a tener que pagarlo.</div>
<div>
<br /></div>
<div>
Estoy bastante seguro de que para resolver los casos ante Defensa del Consumidor tienen que poner a un abogado, probablemente junior, con un costo de unos $ 200 la hora. Y diría además que cada uno de esos casos les requiere entre 10 y 15 horas de trabajo, así que no creo que puedan safar de un costo de entre $ 2000 y $ 3000, por el sólo hecho de que yo vaya a presentar una denuncia.</div>
<div>
<br /></div>
<div>
<div>
Y es que, a estas alturas, además de tener razón en estar enojado y en el reclamo por faltas que voy a hacer, también quiero un poco de sangre de Fibertel arriba de la mesa.</div>
<div>
<br /></div>
<div>
Viejo dicho Klingon: la venganza es un plato que se sirve mejor frío.</div>
</div>
<div>
<br /></div>
<h3 style="text-align: left;">
¿Porqué estoy contando todo esto?</h3>
<div>
<br /></div>
<div>
Como dije al principio, hay varias cosas de esta historia que tienen aplicación directa en como se manejan los negocios, a saber:</div>
<div>
<br /></div>
<h4 style="text-align: left;">
<b>1- La paciencia tiene un límite</b></h4>
<div>
<br />
Cuando un cliente te está dando oportunidades de hacer bien las cosas, no significa que te va a seguir dando oportunidades para siempre. En algún momento tenés que enderezarte y resolver el problema.</div>
<div>
<br /></div>
<div>
Yo se perfectamente que es muy probable que tenga una menor calidad de servicio en Telecentro. De hecho, creo que Fibertel es uno de los mejores ISP que hay en Argentina.</div>
<div>
<br /></div>
<div>
<blockquote style="font-size: 0.9em; margin-left: 2em;">
<i><b>Nota</b>: esa sensación rara que tuviste mientras leías esa última oración es Customer Loyalty quemándose y largando un humo negro que contamina el medio ambiente y daña la capa de ozono.</i></blockquote>
</div>
<div>
<br /></div>
<div>
Cuidá a tus clientes, entendé lo que necesitan y buscá la forma de dárselos (no siempre tiene que costarte algo).</div>
<div>
<br /></div>
<div>
Y <i>tus clientes</i> está usado en un sentido amplio de la palabra. Si trabajás en relación de dependencia, tu jefe es <i>tu cliente</i>. También tiene que estar contento/a (así como tu jefe tiene que tenerte contento/a). Tu valor de mercado está en relación a que tu jefe te siga dando negocio, mejores y diferentes negocios. Tu valor de mercado es tu sueldo. Tu proyección como empresa es tu carrera, y no podés tener una carrera si tus jefes no te dan oportunidades. Eso requiere Customer Loyalty.</div>
<div>
<br />
<h4 style="text-align: left;">
2- Customer Loyalty es un recurso no renovable</h4>
<br />
Crear y hacer crecer una relación comercial (sea entre empresas, por contrato, consultoría o relación de dependencia) es algo que lleva mucho tiempo, pero el daño irreparable se hace rápido.<br />
<br />
Justo como en este caso. Y si leiste con atención la pequeña opereta de historia, habrás notado que el punto de inflexión fue el 15 de Febrero. Ese día, cuando crearon un ticket y me dijeron que tenía que resolverse en 72 horas y que me iban a llamar. Justo en ese momento. Apostaron todo a cara o cruz. La <i>resolución</i> JLC-98902342 es mi inconsciente percibiendo esa apuesta y aceptándola.<br />
<br />
Podrían haberme llamado para decir cualquier cosa (ej: estuvimos, tocamos el portero y nadie atendió, que se yo, podría haber funcionado mal). Cualquier cosa habría entendido.<br />
<br />
Pero no llamaron. Y perdieron la apuesta. Fue en un instante.<br />
<br /></div>
<h4 style="text-align: left;">
3- Hacé bien las cuentas</h4>
<div>
<br />
Las empresas (especialmente las grandes), normalmente clasifican a sus clientes en diferentes conjuntos, que van desde menor <i>valor</i> hasta premium. Ese <i>valor</i> (<i>long term value</i> o <i>valor a largo plazo</i>) tiene que ver con el prospecto de ganancias que la empresa puede obtener de ese cliente (y no te sientas tan ofendido/a, porque a tus amigos/as también los clasificas así y por algunos harías cosas que no harías por otros).<br />
<br />
En mi caso específico con Fibertel, yo no creo haber estado en el segmento super-hiper-top, pero estoy seguro que estaba de la media para arriba. Con Internet 6Mb, DVR (graba), TV-HD + Pack HBO y cada tanto algún consumo Pay Per View. Bastante lejos del cliente que sólo tiene internet y que llama para quejarse y obtener un descuento cada vez que hay un aumento.<br />
<br />
Además de eso, con 4 años de antigüedad, así que los costos de instalación inicial y del DVR (que imagino será de unos U$S 400) ya los tenían más que amortizados.<br />
<br />
Estimo que Fibertel debía estar obteniendo entre $ 150 y $ 200 de ganancia neta mensual de los servicios que me cobraba. Y considerando el tipo de cliente que soy (que busca más su comodidad aunque le cueste un poco más, lo cual implica poca probabilidad de cambio de proveedor), calculo que el <i>largo plazo</i> estaría en el rango de 5 a 10 años.<br />
<br />
Así que (y esto es otra estimación) mi valor de largo plazo como cliente para Fibertel podría calcularse como ($150..200) * 12 * (5..10 años), es decir, entre $ 9000 y $ 24000 en los próximos años. Parece poco, pero multiplicá esto por cientos de miles.<br />
<br />
Esa es la ganancia que Fibertel deja de tener por el simple hecho de que yo me pase a la competencia.<br />
<br />
Pero hay un poco más todavía. El hecho de que yo vaya a presentar una denuncia ante Defensa del Consumidor tiene un costo para ellos, como ya mencioné.<br />
<br />
Es así que lo más probable es que perderme como cliente además les implique perder las ganancias que obtuvieron de mi en el último año!<br />
<br />
Pero espere! Para hacer esta oferta aún más imperdible: como dije, creo que es posible que en algún momento deje Telecentro y vuelva a Fibertel. Pero eso también tiene un costo. Que se llama <i>Costo de Adquisición de Cliente</i> y que la empresa te paga en la forma de promoción (o descuento) por darte de alta. Si es que alguna vez vuelvo a contratar Fibertel, la empresa también va a tener que pagar ese costo.<br />
<br />
<blockquote style="font-size: 0.9em; margin-left: 2em;">
<i><b>Edición pre-publicación</b>: posiblemente pienses que mis números están equivocados, y probablemente lo están. A veces, antes de publicar un post se lo envío a algún amigo para que lo lea y me de su opinión. Bien, en los 50 minutos que llevó obtener esa opinión, volvieron a llamarme del Centro de Gestiones Especiales de Fibertel. Así que sí, creo que mis números son demasiado conservadores.</i></blockquote>
<br />
Así que cuando quieras <i>perder</i> un cliente (o mejor, valorar apropiadamente un cliente), hacé bien las cuentas!<br />
<br /></div>
<div>
<h3 style="text-align: left;">
Conclusión</h3>
</div>
<div>
<br />
Fue bueno mientras duró.</div>
<div>
<br /></div>
<div>
Suficiente para ser mi decimoquinto post (y primero en español, go figure!).</div>
</div>
José Luis Campanellohttp://www.blogger.com/profile/00233954763631710324noreply@blogger.com2tag:blogger.com,1999:blog-2618135660518567589.post-18618783103173269402012-12-24T20:13:00.000-03:002012-12-24T20:13:58.117-03:00Another year is ending. Did you make a donation?<div dir="ltr" style="text-align: left;" trbidi="on">
<br />
Another year. Time goes by faster and faster.<br />
<div>
<br /></div>
<div>
I usually wrap up the year by making some donations.</div>
<div>
<br /></div>
<div>
This time i donated to:</div>
<div>
<br /></div>
<div style="text-align: left;">
<ul style="text-align: left;">
<li>Wikimedia Foundation. The donation page is <a href="http://wikimediafoundation.org/wiki/WMFJA085/en" rel="nofollow" target="_blank">here</a> (english) or <a href="http://wikimediafoundation.org/wiki/WMFJA085/es" rel="nofollow" target="_blank">here</a> (spanish).</li>
<li>The SETI Institute. The donation page is <a href="https://www.teamseti.org/donate" rel="nofollow" target="_blank">here</a>.</li>
</ul>
</div>
<div style="text-align: left;">
<br />
Also, all this year i've been making monthly donations to the World Food Program (<a href="https://www.wfp.org/donate/fillthecup" rel="nofollow" target="_blank">english</a> or <a href="https://es.wfp.org/donate/done" rel="nofollow" target="_blank">spanish</a>) and Nuru International (<a href="http://www.nuruinternational.org/donate/" rel="nofollow" target="_blank">here</a>).</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
So, end this year doing something for others. That is money well spent and it actually makes you feel better about yourself.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
This is enough to be my fourteenth post.</div>
</div>
José Luis Campanellohttp://www.blogger.com/profile/00233954763631710324noreply@blogger.com0tag:blogger.com,1999:blog-2618135660518567589.post-82366172891348719312012-12-24T19:58:00.001-03:002012-12-28T12:59:53.232-03:00How much memory does a JVM need?<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
<i><span style="font-size: xx-small;"><u>Update</u>: fixed some typos pointed out by Ezequiel and Lecko.</span></i><br />
<br />
At this point in IT affairs, we should change the old US saying to go something like "Nothing is certain but death, taxes and a JVM Full GC".</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
Because, no matter what, one thing you can count on is that a JVM will eventualy perform a Full <i>Stop the World</i> Garbage Collection. This is true with Concurrent Mark and Sweep (-XX:+UseParNewGC -XX:+UseConcMarkSweepGC), and with the relatively new G1 garbage collector on Java 6 (-XX:+UnlockExperimentalVMOptions -XX:+UseG1GC).</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
Hence, the rigth thing to do is to give the JVM as much memory as you can afford. Right? Well, not necessarily.</div>
<div style="text-align: left;">
<br /></div>
<h3 style="text-align: left;">
Some ground leveling</h3>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
What follows is in line with the old CMS (Concurrent Mark Sweep) GC because of the memory structure I will discuss. The G1, as far as I know, uses a somehow different structure that is more resilient, but on the end, it will perform a Full <i>Stop the World</i> Garbage Collection, just less frequently.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
The standard memory arrangement for a JVM is to divide the memory in three. You have the New memory, the Tenured memory (OldGen) and the Perm (permanent) memory. The Permanent Memory is used to hold bytecode, class definitions and things like that.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
The New Memory is itself divided in two: the Eden and the Survivor areas. Eden is where short lived objects are created (that is: when inside a method you create a string, an object is created in Eden and then it is destroyed when the method scope is destroyed). This Eden arrangement is what makes for a ligth weight GC. How does it work? Well, if you come to think about it, when you have a server thread, you wait for requests and when you get one, you usualy call a method to solve it. This method creates everything in Eden and when it terminates, almost every object residing in Eden has been destroyed (because of method scoping). So you can mostly reclaim all of Eden's memory without having to think about handling memory fragments. Empirically, most objects in a JVM have a very short lifespan (or high mortality rate, if you want) and Eden is arranged for that.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
Yes, I said <i>almost every object in Eden</i> has been destroyed. That is what the Survivor area is for. Because objects that survive method scoping (ex: the method's result value) will be sent (copied) to Survivor. When Eden is full, then a Minor GC will occur. This Minor GC will do a few things:</div>
<div style="text-align: left;">
<br /></div>
<ul style="text-align: left;">
<li>claim all Eden's dead objects,</li>
<li>move all Eden's live objects to the Survivor area (by copying),</li>
<li>will stage the other Survivor objects (using generations, akin to being X years old), and</li>
<li>send the oldest generation of Survivor objects to Old Generation (in Perm memory) by copying</li>
</ul>
<br />
<ul style="text-align: left;">
</ul>
<div style="text-align: left;">
<i><span style="font-size: xx-small;"><b>Note</b>: there are actualy two survivor areas, that are used alternatively to make things more efficient.</span></i></div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
If you carefully think about what I've just said, you will realize that an object can be dead to the program (meaning: no longer accessible) but still be using memory and pretty much alive for all intended purposes. Objects are not realy destroyed until they are Garbage Collected. This is what the Mark and Sweep does: decide what objects are reachable (alive) and what objects are not (to destroy them). And yes, this is why you should realy realy think twice before using finalizers, because the finalizer is called when garbage is being collected, not when the object gets out of scope. This keeps the resources being used and slows down the GC process because it cannot just kill the object, it has to give the finalizer a chance to run, thus the object survives the current GC cycle and memory is still occupied.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
See the following chart to fully grasp what happens on a Minor GC:</div>
<div style="text-align: left;">
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2bwGiLV0cdY98ZpW_lnjWzCMjZJ87g7o6tSLvcCb4PgUvOlQXIZm58PUrkD6Cr1mNPVgEmsyLHb0i_76UQnTfVuCt1OroRUEQNY7QwMcujUqrhzo_KN9uu9NyQ6mRJeb3J-97fcvc2ROF/s1600/NULL+-+12+-+JVM+Memory+Model.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="216" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2bwGiLV0cdY98ZpW_lnjWzCMjZJ87g7o6tSLvcCb4PgUvOlQXIZm58PUrkD6Cr1mNPVgEmsyLHb0i_76UQnTfVuCt1OroRUEQNY7QwMcujUqrhzo_KN9uu9NyQ6mRJeb3J-97fcvc2ROF/s320/NULL+-+12+-+JVM+Memory+Model.png" width="320" /></a></div>
<br />
The question is: what happens when Old Gen is full? Does the program halt? Enter the Full <i>Stop the World</i> Garbage Collection.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
When this happens, the GC stops all processing in the JVM and starts collecting memory on zombie objects (alive but unreachable). This encompases objects in the Old Gen, as much as objects on Eden, Survivor and things in Perm memory. The Perm is included because some loaded or dinamicaly created classes might no longer be needed and that memory could be claimed.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
As for what goes on with the rest of objects, well, they are marked as dead or alive and the alive objects are <b>memory copied</b> to new positions in order to create a large block of unused memory. This <b>copying</b> is expensive on it's own, but what makes the thing even worse is that every pointer (reference) to the moved object must be also updated to point to the new object's address. And this is why the <i>Stop the World</i>, because we can't be changing things while the rest keeps moving (see the section at the end about <a href="http://nullpointerassign.blogspot.com.ar/2012/12/how-much-memory-does-jvm-need.html#azul-zing">Azul's Zing JVM</a>).</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
<i><span style="font-size: xx-small;"><b>Note</b>: this pointer fixing can be really taxing on the processor's L2/L3 cache lines, depending on how spread in memory are the pointers to be fixed and how many times the object is pointed to, so a very linked structure (ex: a highly connected graph) will probably slow down the process.</span></i></div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
From all this, it follows that the amount of time it takes to do a Full GC is largely dependent on the number of objects still alive that must be copied, which largely depends on the application logic and the amount of memory to scan (AKA: the Old Gen memory size).</div>
<div style="text-align: left;">
<br /></div>
<h4 style="text-align: left;">
Some Caveat</h4>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
It is custom practice to set the memory sizes by specifying the same value to two parameters. The JVM accepts a <i>Max</i> size and a <i>Start</i> size. If you set <i>Start</i> to less than <i>Max</i>, then the JVM allocates <i>Size</i> and grows when needed. This growing (and shrinking) is expensive, so most people usually sets <i>Max</i> and <i>Start</i> to the same value. For example:</div>
<div style="text-align: left;">
<br /></div>
<ul style="text-align: left;">
<li>the Perm size can be set to 512Mb set with <span style="font-family: Courier New, Courier, monospace;">-XX:PermSize=512m -XX:MaxPermSize=512m</span></li>
<li>the Perm total size can be to 8Gb set using <span style="font-family: Courier New, Courier, monospace;">-Xms8000m -Xmx8000m</span></li>
<li>the New total size can be set to 128Mb with <span style="font-family: Courier New, Courier, monospace;">XX:NewSize=128m -XX:MaxNewSize=128m</span></li>
</ul>
<br />
<ul style="text-align: left;">
</ul>
<h4 style="text-align: left;">
How to get information on Garbage Collection</h4>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
In order to know what is going on at the GC level, you can add the following parameters to your JVM on startup:</div>
<blockquote class="tr_bq" style="text-align: left;">
<span style="font-family: Courier New, Courier, monospace;">-XX:+PrintGCTimeStamps</span><span style="font-family: Courier New, Courier, monospace;">-XX:+PrintTenuringDistribution</span><span style="font-family: Courier New, Courier, monospace;">-XX:+PrintGCDetails</span><span style="font-family: Courier New, Courier, monospace;">-Xloggc:<file path<path a="a" file="file" to="to">></path></span></blockquote>
<div style="text-align: left;">
Also, you can use the <i>jstat</i> command to see what's going on in real time on a running JVM (it requires <i>jstatd</i> and probably some configuration settings). The command takes the form:</div>
<blockquote class="tr_bq" style="text-align: left;">
<span style="font-family: Courier New, Courier, monospace;">jstat -gcutil <pid<pid>> 1s 1000</pid></span></blockquote>
<div style="text-align: left;">
where 1s is 1 second intervals and 1000 is the number of samples to take. Full jstat command line arguments for Java 5 can be found in this <a href="http://docs.oracle.com/javase/1.5.0/docs/tooldocs/share/jstat.html" rel="nofollow" target="_blank">link</a>.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
This command is interesting because it shows memory usage in percentages and also the number of Minor and Full GC performed on the system with the total accumulated time used for each type of GC. You can then use this to figure out how long it takes to perform each type of Garbage Collection.</div>
<div style="text-align: left;">
<br /></div>
<h3 style="text-align: left;">
So, how much memory does a JVM need?</h3>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
This actualy depends, as it always do. There is, of course, a minimum below which your application will not even start. Obviously, you need more than that minimum.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
Now, if you are a <i>lucky bastard</i>, then you probably can get away with having NO Full <i>Stop the World</i> GC. Yes, I said before that you can count on death, taxes and a Full GC. So, how is it possible to getting away with no Full GC?</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
Well, it has to do with the definition of <i>lucky bastard</i>. You fall into that category if you can manage to satisfy these two conditions:</div>
<div style="text-align: left;">
<br /></div>
<ol style="text-align: left;">
<li>have an application activity cicle that has a very low activity period. For example: you serve customers in just one country and you have near zero activity at 3 am.</li>
<li>have an application that, measured between two adjacent low activity cicles and without a Full GC, consumes less memory than your server's real memory. For example: between today at 3 am and tomorrow at 3 am, your app requires 4 GB total memory and your server has 8 GB phisical memory.</li>
</ol>
<div style="text-align: left;">
<i><u>Update</u>: i will clarify on rule 2. Let's say you realize the JVM executes a Full GC every hour and that GC releases 1Gb of OldGen memory every run. Then, on a 24 hour cycle, that JVM will require 24Gb of memory, plus the minimum required memory, plus some extra memory for safety to avoid executing a Full GC. If your server has 48Gb of memory, then allocating, say, 32Gb to the JVM should do the trick.</i><br />
<br /></div>
<div style="text-align: left;">
In other words, <i>lucky</i> means that you can allocate 6 GB of memory to your JVM (which avoids a Full GC during working hours) and you also restart your app at 3 am, before the Old Gen gets full.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
Yes! You get away without a Full GC because you kill your JVM before it actualy needs it. This approach is being used by many finnancial institutions (specially high frequency traders) to avoid the latency of a <i>Stop the World</i> GC.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
Now, if you are an average mortal, your JVM will actualy stop to GC at some point or another. So what's the size to use for the Old Gen? It depends on your processing requirements and requires trial and error. What you must embrace is the fact that the Full GC <b>cannot be avoided</b>. What you can do, however, is to <u>control</u> for how long the Full GC stops the world. As said, that depends on the total number of object, which in turns depends on the total memory that the GC must scan and copy.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
So the value that you must set can actualy be tuned to your needs, but the unecpected thing is that the memory size might have to go down for you to hit your target SLA requirements (ex: all requests must take less than a second to be responded).</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
I know this is counter intuitive, and frigthening. A few days ago, i was talking to a customer and said that the total configured memory a JVM had should probably go down from 4.5Gb to 3Gb to reduce the Full GC stop time from 2.5 seconds to less than 1s. A few eyebrows were raised and some fearfull looks crossed the room.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
Nobody is happy with the idea of reducing the allocated memory to a process, specially in these days of Moore's law when cheap translates into a default of <i>more is better</i>.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
But the GC algorithm is time linear on memory size and perhaps it's better to have frequent shorter stops than less frequent but longer stops. Your SLA might call for this.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
You see, sometimes <i>less is better</i>.</div>
<div style="text-align: left;">
<br /></div>
<h4 style="text-align: left;">
So, what is your target Memory value?</h4>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
Well, it's a trial an error, but now you know how to tune the value. You check how frequent is the Full GC and how long it takes and figure out if you are a <i>lucky bastard</i>. If you are, then all set.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
If you are not, then the duration of the Full GC should tell you if you have to add or remove Old Gen memory and then tune the size slowly until you get the expected results.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
There is also some good news to those <i>lucky bastard</i> wannabes. If you are elastic on the number of CPU/Cores/memory available, then you can get away with no Full GCs, even if your app has significant activity all day long. All you have to do is monitor when your JVMs are approaching the Full GC mark and restart them. An Old Gen at 95% utilization can mark a good time when you recycle the JVM, but this also needs tuning.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
Whether you are tuning the NewMem size or the OldGen size, the two most important things to remember are:</div>
<div style="text-align: left;">
<br /></div>
<ul style="text-align: left;">
<li>that the value you set has direct impact on the duration of the GC cycle, and</li>
<li>that the type of garbage created depends on your applications, so the values for one app will not necessarily work for a different app.</li>
</ul>
<div style="text-align: left;">
<br /></div>
<h4 style="text-align: left;">
What about Application Logic?</h4>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
As i mentioned before, two factors affect the Full GC time. The first one is the memory size which we just covered.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
The second factor is Application Logic. More precisely, the reference structure and the pattern of change of Long Lived objects.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
The more intricate the links between objects, the more expensive it becomes to move an object around in memory, due to the pointer fixing. So you probably don't want a big mesh of objects referring to each other. A simple reference structure is probably the best choice here, unless you absolutely need something more complex.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
The other issue to control is the pattern of change. It will generally be best if your long term objects (ex: cached options and configuration) are mostly stable over long periods of time. The reason being that a destroyed long term object causes a hole in memory that must be reused, which in turn requires moving around other long term objects.</div>
<div style="text-align: left;">
<br /></div>
<h3 id="azul-zing" style="text-align: left;">
Are we stuck with Full GC?</h3>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
There is also a bit of other good news, if you can afford to spend some green currency bills.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
There is a company called <a href="http://www.azulsystems.com/" rel="nofollow" target="_blank">Azul Systems</a> that created a special brand of JVM called <a href="http://www.azulsystems.com/products/zing/whatisit" rel="nofollow" target="_blank">Zing</a>. One of the biggest features of Zing is that it has a GC that makes no Full <i>Stop the World</i> GCs.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
I haven't used it and I have no relation with Azul Systems, so i can really say much more about it, but it sure sounds cool.</div>
<div style="text-align: left;">
<br /></div>
<h3 style="text-align: left;">
Some extra info</h3>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
If you want to know more about Garbage Collectors, you can watch this <a href="http://www.infoq.com/presentations/Java-GC-Azul-C4" rel="nofollow" target="_blank">InfoQ presentation</a> by Gil Tene (CTO and co-founder of Azul Systems). It's about Garbage Collectors in general. Again, I have no relation to Azul Systems in any way, but the information on this presentation is very good.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
This is enough to be my thirteenth post.</div>
<div style="text-align: left;">
<br /></div>
</div>
José Luis Campanellohttp://www.blogger.com/profile/00233954763631710324noreply@blogger.com0tag:blogger.com,1999:blog-2618135660518567589.post-91165177142422133622012-08-27T09:00:00.000-03:002012-08-27T09:00:00.764-03:00Harder to kill that cockroaches<div dir="ltr" style="text-align: left;" trbidi="on">
About 300 millions of years ago, <a href="http://en.wikipedia.org/wiki/Cockroach" target="_blank">cockroaches</a> appeared on Earth as a life form. I'm sure the first ones were very primitive (as it is with every life form), but at some point one specific kind of cockroach evolved, and it took over the Earth, because it <i>was the most advanced cockroach</i> that ever existed! This one was extremely resilient to almost everything (even radiation) and will probably outlive us all. Some variations of this insect evolved later (there are 4500 sub-species, 30 of which are related to human habitats, with specific variations by continent).<br />
<br />
In 1990, the first <a href="http://en.wikipedia.org/wiki/Web_browser" rel="" target="_blank">web browser</a> came to exist. It was called "World Wide Web". I'm sure it was primitive (compared with what we have today) and it spun a whole lot of new experiments. Soon we had subspecies like NCSA Mosaic (and i'm old enough to have used it), then Netscape (which i've also used), Internet Explorer, Firefox, Safari, Chrome, Opera and many many others. Not sure there are thousands of browser subspecies, but i would say there are at least a hundred, easily.<br />
<br />
You probably see were this is going, but in August 2001, the <i>most advanced</i> species of browser appeared on Earth, and it took over the planet. It was Internet Explorer 6. There existed many subspecies (different service pack level, different language, different configuration and enabled options). Some of these subspecies were geography specific. For example, I understand that South Korea has a government mandated authentication protocol that was supported only by IE 6. There is also the chinese version of IE 6 which is used in... China.<br />
<br />
IE 6 inherited an adaptation from it's previous primitive sibling: <a href="http://en.wikipedia.org/wiki/AJAX" target="_blank">AJAX</a>. It took it hidden in its DNA until it became a real evolutionary advantage, circa 2005, when evolution in behavior began, in the form of innovation. And it was a boom. It learned so many new tricks in a few years that it changed the environment forever, but it was not very well suited to this new environment. Evolution took over again, with Firefox, IE7, Chrome, etc.<br />
<br />
When it first appeared, it was the most advanced on Earth... exactly 11 years ago, on August 27, 2001.<br />
<br />
But 11 years in the computer industry is like some number of millions of years in evolution, and the damn thing is still lingering on, like a cockroach that refuses to die. When support was finally discontinued, Microsoft itself declared that IE 6 is a fossil, but it's a living fossil.<br />
<br />
Not something of the past. A living fossil! Something you would expect in a Hollywood movie like Journey to the Center of the Earth. Or maybe on a Steven Spielberg movie where some scientist recovers a portion of bits from an old damaged CD-ROM and recreates it using some bits from another application.<br />
<br />
In an effort to eradicate the thing, Microsoft created the <a href="http://www.ie6countdown.com/" target="_blank">IE6 Countdown</a> site, to track where on the planet this species is still a pest, carrying deadly deseases to developers, businesses and users alike.<br />
<br />
Still 6% presence, mostly in China, were it has 21% market share! I have been tracking the evolution of the fight and I'm happy to say that from June 2012 to July 2012, Argentina (where i live) crossed the barrier from 1.2% market share to less than 1% (0.9%, to be precise). Argentina it's now marked green on the map!<br />
<br />
But is a die-hard. Even Norway (that crossed the 1% barrier long ago) can't still get rid of it, with 0.1% market share.<br />
<br />
So, what can you do? When you see a cockroach in your kitchen don't you kill it? Do the same thing with IE6. When you see someone using it, kill it. Go to the web, search for another (any) browser, download it, install it and feel good you did a good thing for that someone.<br />
<br />
IE 6 did good. Changed the environment to something more hospitable. Enabled an explosion. But is now past its extinction time.<br />
<br />
This is enough to be my twelfth post.</div>José Luis Campanellohttp://www.blogger.com/profile/00233954763631710324noreply@blogger.com3Buenos Aires, Autonomous City of Buenos Aires, Argentina-34.6037232 -58.3815931-34.7082817 -58.5402081 -34.499164699999994 -58.222978100000006tag:blogger.com,1999:blog-2618135660518567589.post-91687738312927333712012-03-30T19:22:00.002-03:002012-03-30T19:22:49.188-03:00DB Connection pool loosing connections (AKA: the not so transparent firewall)<div dir="ltr" style="text-align: left;" trbidi="on">
I'm posting this because i've seen this problem many times, specially in corporate environments.<br />
<br />
<div style="text-align: left;">
<b><span style="font-size: large;">The symptom</span></b></div>
<br />
<div style="text-align: left;">
You have an application (usually a java app running on an app container such as tomcat) that connects to a (any) database. You change from using a simple "connect to the db when you need it" to "use a connection pool to eliminate connection setup time".</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
You immediately rollback the connection pool change because the connections seems to die after some time (usually low usage or night time), making your application unresponsive.</div>
<br />
<b><span style="font-size: large;">The explanation</span></b><br />
<br />
Well, there are many possible causes for the symptom, but one of the most common causes i've seen (specially in corporate environments) is due to the widespread use of <i>transparent firewalls</i>.<br />
<br />
<span style="font-size: x-small;"><i><b>As a side note</b>: be wary of anything that calls itself a transparent something, such as transparent firewalls or transparent proxies because they are, at best, translucent.</i></span><br />
<br />
To understand the problem, we must first understand that the TCP stack closes a connection after it interchanges a number of packets with it's peer TCP layer. The message flow is complex because a TCP connection can be half closed, meaning one of the peers is done with it but the other is still sending. In order to close a connection, one of the peers can send a message with the FIN flag set (which triggers the FIN-WAIT statuses you can see with the <i>netstat</i> command). The FIN-WAIT status then progresses to FIN-WAIT-2, then to CLOSING and then to CLOSED.<br />
<br />
Another alternative to connection closing is for one peer to send a TCP RST (reset) packet, indicating that the whole connection is in failure mode and shall be terminated.<br />
<br />
TCP keep-alive was invented so that long lived connections could be kept alive. This includes exchanging ACK (acknowledge) packets between peers every now and then (usually configurable at the kernel level).<br />
<br />
When keeping alive a connection, if a host doesn't hears from it's peer, then a <b>lengthly</b> series of attempts to reach the peer begins (we'll come back to this later). Upon failure, this series ends with the connection in the TIME-WAIT status that occupies a connection table slot for quite some time.<br />
<br />
Enter the corporate firewall.<br />
<br />
The most common usage of a corporate firewall is to actually do <i>connection filtering</i>. This means: telling what host can connect to what other host+port. I'm not really sure why a firewall is used for this, because most basic routers (and even switches) can do this kind of thing, albeit more in an NAT kind of way, not in an ip-hiding way.<br />
<br />
Connection filtering works something like this:<br />
<br />
<ul style="text-align: left;">
<li>When a connection is attempted, the firewall looks at the filtering rules to check if the 4-uple <i><src-ip, dst-ip,="" dst-port="" src-port,=""></src-ip,></i> is in the list of allowed connections.</li>
<li>If it's not on the list, then <b><u>one</u></b> of the following happens:<br />
<ul>
<li>an RST packet is sent to the source host, blocking the establishment of the connection. Not the best option, as it informs the source host that the remote ip does exists but doesn't like the port.</li>
<li>nothing is sent, letting the remote host retry until it timeouts. This is good, as the source host does not know the remote ip is valid, preventing address/port network scans.</li>
</ul>
</li>
<li>If the 4-uple is on the list, then <b><u>both</u></b> of the following happen:<br />
<ul>
<li>An entry representing the 4-uple is created in an in-memory table inside the firewall software.</li>
<li>The connection setup packet is forwarded to the destination host (oftentimes with some IP address changing).</li>
</ul>
</li>
</ul>
<br />
So far so good.<br />
<br />
Now, for a reason that escapes me, many (most?) corporate firewalls don't like low traffic long lived TCP connections. They don't like these connections badly enough to go to the extreme of detecting TCP keep alives in order to be able to euthanize these connections.<br />
<br />
Low traffic long lived connections in corporate environments are darwinianly selected out by corporate firewalls. This natural selection is what's causing trouble in your environment.<br />
<br />
The real problem is that the connections are not being terminated, just being euthanized. Yes, there is a big difference.<br />
<br />
In a connection termination scenario, the firewall has to do two things:<br />
<br />
<ol style="text-align: left;">
<li>Send each real peer a TCP RST packet. This gives the peers a notification of the massacre. This notification enables the host's kernel to return an error to the application which will notice the connection is closed and be able to re-connect.</li>
<li>Drop the 4-uple in the firewall valid connections table (thus blocking all following traffic).</li>
</ol>
<br />
But the connection is euthanized. This means the firewall usually does just step number 2.<br />
<br />
The peers, not knowing about the transparent firewall, enter the lengthly process of trying to reach out each other. A process that is doomed to fail because the firewall is not forwarding the traffic.<br />
<br />
Eventually, the (kernel) peers decide to give up and close the connection, letting the application know with an error. The key word in the previous sentence being <i>eventually</i>. In the mean time, the connections are zombies inside the kernel and the application is not notified until it's too late.<br />
<br />
<span style="font-size: large;">The Solution</span><br />
<br />
The good news is that you can probably fix this problem without trying to let the people running the firewalls understand they are causing trouble and change defaults (after all, this is a silo'd corporate environment, what are your chances of effecting change on a different department?).<br />
<br />
If you are using a standard connection pool, there are <b><u>two</u></b> options you have to set to avoid triggering the euthanasia:<br />
<br />
<br />
<ul style="text-align: left;">
<li>Set the timeout (or keep alive) connection time to a lower value than the firewall's idle value. If the guys running the firewall don't tell you what this value is, you can always hand test it.</li>
<li>Set the option to do an active keep alive. This will effectively send a command to the database causing real traffic to go thru the connection. This involves a command without side effects (such as <i>select 1 from dual</i>, <i>select current_time</i> or something like that).</li>
</ul>
<br />
If you are not using a connection pool but an ad-hoc application-level protocol, you can always create a nop (no operation command) that responds some data.<br />
<br />
<br />
This is enough to be my eleventh post.</div>José Luis Campanellohttp://www.blogger.com/profile/00233954763631710324noreply@blogger.com0tag:blogger.com,1999:blog-2618135660518567589.post-60451839625792286892012-03-08T00:27:00.000-03:002012-03-08T00:27:31.004-03:00Will extreme poverty ever end? Nuru International<div dir="ltr" style="text-align: left;" trbidi="on">
Hi,<br />
<br />
after watching a <a href="http://www.youtube.com/user/GoogleTechTalks">Google Tech Talk</a> by Jake Harriman, founder and CEO of Nuru International (<a href="http://nuruinternational.org/">nuruinternational.org</a>), titled <a href="http://www.youtube.com/watch?v=R4wFkm1EiM4">Will Extreme Poverty Ever End?</a>, three things got into my mind.<br />
<br />
<div style="text-align: left;">
The very <b>first</b> one was to actually go and <a href="http://www.nuruinternational.org/donate/">make</a> a monthly donation. Go ahead and do so yourself. Not only you will never die of a severe case of donation, it also increases your overall happiness, which reduces stress. So go and do something good for others and yourself (or yourself and others).</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
The <b>second</b> thing i couldn't stop, is wondering about this guy. A US Marine that went to Afghanistan (something that some people, including me, considers to be one of those bad things the US do abroad). This guy decided to keep fighting terrorism on very different front lines: those of NGOs and humanitarian relief.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
I can't help to think that when a soldier decides that she or he will be more effective fighting terrorism by eradicating poverty, then a lot of good things are going on in the world. Many many more good things that we would assume if we just listen to mainstream news organizations, or old folks saying things used to be better.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
While i never had the <i>paralysis problem</i> when facing a decision of letting go of things and start anew, it is very telling that changes like the one Jake Harriman did are not only possible, but feasible.</div>
<div style="text-align: left;">
<br />
Bold move.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
The <b>third</b> thing that got into my mind is innovation. You can tell that this guy took a lot of bagage from the Marines (well, you can't stop being who you are) and applied it to the field of humanitarian relief. Nuru works like a contractor. They actually don't try to reinvent the weel, but to figure out the most effective weel for each type of task. Nuru partners with other NGOs so that both NGOs can better fulfill their respective missions. And Nuru is trying to actually measure things to figure out what works and what doesn't, to try new things and fail fast.<br />
<br />
A very agile style.<br />
<br />
I couldn't help to compare this approach with the Record Labels's and Entertainment Industry's. We recently were bombarded with the whole SOPA/PIPA debate (even here in Argentina was all over the news). These industries don't try new things. They don't even like others to try new things. In the case of Hollywood, they even disregard they own history. Independent studios relocated to the US west coast because of the many patents and laws on the east coast were stifling innovation and new business models. And these independent studios grew to be Hollywood.<br />
<br />
So, instead of collaborating with Napster, Labels shut it down. Neither got the chance to better fulfill their respective missions.<br />
<br />
Nothing like contrast.<br />
<br />
This is enough to be my tenth post.</div>
</div>José Luis Campanellohttp://www.blogger.com/profile/00233954763631710324noreply@blogger.com0tag:blogger.com,1999:blog-2618135660518567589.post-74557816180488216352011-12-29T14:54:00.000-03:002011-12-29T14:54:27.023-03:00Donate to Wikimedia Foundation for 2012 Fund Raising<div dir="ltr" style="text-align: left;" trbidi="on">
<br />
Once again, a year is about to end.<br />
<br />
As i did last year, i've already donated money to the Wikimedia Foundation.<br />
<br />
Last year, Wikimedia Fundraising lasted 50 days and got to cover the expected 15 million USD.<br />
<br />
This year, Wikimedia is expecting to raise 28.3 million USD. Donations will be accepted through January 2012.<br />
<br />
Do you use Wikipedia? Then <a href="https://donate.wikimedia.org/w/index.php?title=Special:FundraiserLandingPage&country=AR&uselang=en&utm_campaign=20111228EM2.en.AR&utm_medium=email&utm_source=email2.USD.50">donate</a> now!<br />
<br />
Human knowledge should be a public good, like the environment.<br />
<br />
Wikipedia does just that: make knowledge available to everyone that wants it. Please <a href="https://donate.wikimedia.org/w/index.php?title=Special:FundraiserLandingPage&country=AR&uselang=en&utm_campaign=20111228EM2.en.AR&utm_medium=email&utm_source=email2.USD.50">help</a>!<br />
<br />
This is enough to be my ninth post.<br />
</div>José Luis Campanellohttp://www.blogger.com/profile/00233954763631710324noreply@blogger.com0tag:blogger.com,1999:blog-2618135660518567589.post-57298216382652486042011-10-15T19:48:00.001-03:002011-10-15T19:48:34.703-03:00Dennis Ritchie passed away<div dir="ltr" style="text-align: left;" trbidi="on">
Dennis Ritchie was found dead on his apartment last Wednesday (Oct 12). He apparently died last weekend.<br />
<br />
<div style="text-align: left;">
Dennis Ritchie created the C programming language and co-created Unix with Ken Thompson. He also worked with Brian Kernighan.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
This is a very sad October.</div>
</div>José Luis Campanellohttp://www.blogger.com/profile/00233954763631710324noreply@blogger.com0tag:blogger.com,1999:blog-2618135660518567589.post-68045069375837660632011-10-05T21:50:00.000-03:002011-10-05T21:50:21.643-03:00Steve Jobs passed away<div dir="ltr" style="text-align: left;" trbidi="on">After a little over nine months of not writing on this blog, i'm very sad the reason for this post is the passing of <a href="http://www.apple.com/">Steve Jobs</a>.<br />
<br />
Much can be said about the good and bad (of Apple, the philosophy, the technology, his character, etc).<br />
<br />
None of that matters to me today. I feel really sad.<br />
<br />
Having lost family to cancer, i only hope he didn't suffer much.<br />
<br />
</div>José Luis Campanellohttp://www.blogger.com/profile/00233954763631710324noreply@blogger.com0tag:blogger.com,1999:blog-2618135660518567589.post-31627467217570861412010-12-31T17:15:00.000-03:002010-12-31T17:15:34.943-03:00Donate to Wikimedia Foundation!The year is about to end (here in Argentina it's still 2010) and the <a href="http://wikimediafoundation.org/">Wikimedia Foundation</a> is doing another round of fund raising to sustain next year's operations. They need only 16 million US dollars (they have 15.2 M already).<br />
<br />
Do you use Wikipedia? The <a href="http://en.wikipedia.org/">english</a> version? The <a href="http://es.wikipedia.org/">spanish</a> version?<br />
<br />
I have donated already. In the process, i discovered that there is an <a href="http://wikimedia.org.ar/">Argentine chapter</a> of the Wikimedia Foundation (not every country has one).<br />
<br />
You can donate <a href="http://wikimediafoundation.org/wiki/Donate">here</a> (english) or <a href="http://wikimediafoundation.org/w/index.php?title=WMFJA026/es">here</a> (spanish). You can also <a href="http://wikimedia.org.ar/wiki/Portada">donate to the local chapter</a> (see the bottom of the page).<br />
<br />
You can check how much money they have already raised by just going to the main wikipedia's page for any language. There is a detailed fund <a href="http://wikimediafoundation.org/wiki/Special:FundraiserStatistics">raising page</a>, yet it seems to have different information (out dated?).<br />
<br />
I hope you all have a happy 2011 and, if you didn't donate in 2010, consider making a donation as one of the first actions of the new year.<br />
<br />
This is enough to be my sixth post.José Luis Campanellohttp://www.blogger.com/profile/00233954763631710324noreply@blogger.com0tag:blogger.com,1999:blog-2618135660518567589.post-45456427019056337092010-12-20T19:42:00.000-03:002010-12-20T19:42:50.377-03:00About (abuse of) PowerThis is an unusual post as it started with something unrelated to IT but i think recent events made it relevant to our industry because of the Java TCK licensing <a href="http://en.wiktionary.org/wiki/kerfuffle" target="_blank">kerfuffle</a> (AKA: Oracle versus ASF).<br />
<br />
A few months ago, i was having lunch with some friends and we were talking about power in politics. Abuse of power in politics, to be specific. After all the talking, i realized that we tend to believe power has properties that it actually doesn't have.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://www.istockphoto.com/stock-photo-9033900-forklift-with-bricks.php" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img alt="Idea of Power - Forklift carrying bricks" border="0" height="313" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh55UDfzp1KeMVFvvLtz0DuDsdPgecvXWfamXC4wS0YDjXEYV7ruC1ooTql4wdYO03-q4_wcJpNO_FWqTP9MGqXUkBoj90x1nSsBeYCa59-TLcrXRa8luwkQlZFXr7wC3Qr_IpkoU39P5FJ/s320/IdeaOfPower.jpg" width="320" /></a></div>We tend to think power is a solid. Something like bricks.<br />
<br />
The idea being that we can get as much power as we can pile bricks.<br />
<br />
Such was the mental image i had when the conversation started, but the image changed rapidly.<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://www.sxc.hu/photo/581041" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img alt="Reality of Power - Hand holding some sand" border="0" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgMLXiclAdBr_MAeHybgvK0mj9rqPRQQdljirvqSJl5FMqilDCBl03Nkazf2cA0khfOSVByoATcLtsEXclZPdK6kpOE8jXYnU87OsoAeXoArHLMj6wQ5JWCB8kzbFfHoOeVU9x7MzBph9tA/s320/SomePower.JPG" width="320" /></a></div>I figured that comparing power with a solid was not entirely right. I was missing something.<br />
<br />
I think power is more of a fine grained solid. To continue the construction metaphor: power is like sand.<br />
<br />
You can have some small amount of power and hold on to it, the same way you can do with sand.<br />
<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://www.istockphoto.com/stock-photo-11162600-trying-to-hold-sand.php" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img alt="Too much power - Hands pouring sand" border="0" height="257" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhe6I-TaDUo-0pcuQ8039wolf2ztRZDcWBN-nPnzuDhC6jKTwcRyHMZsE0rgFcf_AGTVH_a80eDhjD4RhQmclXNKXzJ90QatAnLTUqbcqoyH_i1sBnHcW_uXIt0osT1kTg3m5mE98rBoRDr/s320/TooMuchPower.jpg" width="320" /></a></div>You can even succeed and get a lot of power for a while.<br />
<br />
Yet, as it happens with sand, power filters out eroding while it filters, making the leak bigger and bigger.<br />
<br />
And as you see the leak you begin to get desperate because you are loosing what you hold dear. And you start doing stupid things. Just as anybody does when they have lots of power.<br />
<br />
<div><br />
</div><div>This has happened to genocidal maniacs, dictators, presidents, political parties, police, mobsters, gangsters, companies, CEOs, bosses, abusive husbands, abusive wives, child molesters, bad teachers, bullies, big brothers, little brothers, etc.<br />
<br />
We all have abused power in some situation or another.<br />
We all know abuse comes back to balance the score.<br />
<br />
I think we have recently seen an example of abuse of power in the way Oracle used Java's ownership. My guess: as a way to get Google to pay for using Java on Android (J2ME enabled phones probably pay royalties of some sort). I don't have anything against Oracle making money, yet i personally disagree with this move because it gets the open source programming community and the <a href="http://www.apache.org/">ASF</a> in the middle of a corporation war.<br />
<br />
Oracle may succeed in the short term in getting some more money, but the forces propelling open communities will route around this issue and when that happens, this will undoubtedly come back to bite Oracle on the hand.<br />
<br />
Abuse of power has happened before and will happen again.<br />
Loss of that power has happened before and will happen again.<br />
No matter who. No matter why.<br />
<br />
<i>Disclosure: i'm not active in any open source community.</i><br />
<br />
This is enough to be my fifth post.</div><div><br />
PS: photographs are from <a href="http://www.sxc.hu/">stock.xchng</a> and <a href="http://www.istockphoto.com/">iStockPhoto</a></div>José Luis Campanellohttp://www.blogger.com/profile/00233954763631710324noreply@blogger.com0tag:blogger.com,1999:blog-2618135660518567589.post-48495524448783899642010-10-12T01:08:00.000-03:002010-10-12T01:08:05.070-03:00(not) Messing up your Data Model because of your ORMA few months ago, one of my clients asked me to take a look at the Data Model of one of his applications because of performance problems. A review showed database tables had no indexes, that being the cause of the performance hit.<br />
<br />
Yet a deeper and complex structural problem surfaced upon further inspection. This second problem was, and still is, the result of lack of knowledge of both Databases and ORMs, so i thought it was worth sharing.<br />
<br />
Refactoring an application usually isn't a big deal by itself. But when you have a Database involved, refactoring turns into a very complex problem beyond the realm of a single programmer, because data needs to be restructured. Moreover, when data is comprised of millions of records and over a terabyte of storage, the restructuring process can take months making it impractical and even forbidden by laws and regulations, as it is for this particular case.<br />
<br />
<span class="Apple-style-span" style="font-size: x-large;">The Application and the Implementation</span><br />
<br />
The <i>troubled</i> application stores transactions in a database. It consists of a web service writing to the database and a web interfase to query inserted records. Transactions are composed of primary meta-data and three optional groups of data: Extra meta-data, Business data and LowLevel data. Each four of the data groups are stored on their own table. Tables hold historic information because it must be available for querying at any time and because regulations require it.<br />
<br />
When development started, an Object Model was created for the <i>write web service</i> that could be described as:<br />
<blockquote><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">Four classes to model a transaction:</div><ul><li>class Transaction: holds primary meta-data</li>
<li>class Extra: holds extra meta-data</li>
<li>class Business: holds business data</li>
<li>class LowLevel: holds low level data</li>
</ul><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">These classes are <a href="http://en.wikipedia.org/wiki/POJO" target="_blank">POJO</a> objects; and there is also a DAO class to abstract database access. The classes Extra, Business and LowLevel are attributes of the class Transaction.</div></blockquote>Here is when things started to run amok and reality and implementation started to diverge.<br />
<br />
<i><b>Note</b>: remember Euclidean Geometry? What happens when even a very small divergence between two rect lines is propagated to a very large distance?</i><br />
<br />
Let's see first how the Good Data Model should look like:<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgAvB596Ejo1SsAqfetJkV1Tb3xNFdwHCvizfcP4YIkZZgZ_wNC2zKzqgJu6ujwwbrGsOHXNTtICM5olQLzcH5dw5NQ2jYDPHWrIoI9zmGp5NIgJK9RK8eldwAoJM6VMVKqPQIK0mP6yEjv/s1600/DataModelSmall.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;" target="_blank"><img alt="The Data Model" border="0" height="306" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgAvB596Ejo1SsAqfetJkV1Tb3xNFdwHCvizfcP4YIkZZgZ_wNC2zKzqgJu6ujwwbrGsOHXNTtICM5olQLzcH5dw5NQ2jYDPHWrIoI9zmGp5NIgJK9RK8eldwAoJM6VMVKqPQIK0mP6yEjv/s320/DataModelSmall.PNG" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The Good Data Model</td></tr>
</tbody></table>As the image shows, there should be a table named Transaction, with a monotonically increasing primary key (an auto-incremented id, mapped for example to an Oracle Sequence or an MS SQL Server IDENTITY). There should also be tree satellite tables (Extra, Business and LowLevel), each with just a primary key and, if you feel like it, a foreign key to the Transaction primary key. Notice that this design preserves the fact that satellite tables are dependent upon the primary table. This design is guided by what is called Normal Form in Database courses at colleges.<br />
<br />
However, the implemented Data Model, we will call it Bad, looks like the following:<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjMxcQYYV6xzUM_EJtNyG6SHPgQlQqJjP1XQC-ROBoYHg7QlhXyaH1Us0ivle9Ibs0IxN5RAYU81S4aN3p4uDWTTaywjYI3QV2osEJe74HlNFHg28-yVe1Q9gXjtDAgCWULhQCdahjHhlsq/s1600/DataModelImplementedSmall.png" imageanchor="1" style="margin-left: auto; margin-right: auto;" target="_blank"><img alt="The Data Model" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjMxcQYYV6xzUM_EJtNyG6SHPgQlQqJjP1XQC-ROBoYHg7QlhXyaH1Us0ivle9Ibs0IxN5RAYU81S4aN3p4uDWTTaywjYI3QV2osEJe74HlNFHg28-yVe1Q9gXjtDAgCWULhQCdahjHhlsq/s320/DataModelImplementedSmall.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The Bad Data Model</td></tr>
</tbody></table><div class="separator" style="clear: both; text-align: center;"></div>As the image shows, now each table has an IDENTITY primary key. There are also three attributes on the Transaction table, each having the value of the primary key for the record in the corresponding satellite table.<br />
<br />
I believe two separate things conspired to make the programmer create this Bad Data Model:<br />
<div><ul><li>The Object Model makes the four classes (types) independent of each other. An instance of each class can exist on its own as you can call operator <i>new</i> on that class. Even when this makes syntactic and semantic sense for the Object Model, it doesn't reflect reality and it doesn't make syntactic nor semantic sense for the Data Model.</li>
<li>Not knowing the full set of capabilities of Hibernate ORM. When asked, the programmer's answer was that Hibernate cannot map things like the Good Data Model, even when it made sense from the Database point of view. Of course, Hibernate can properly map the right Data Model (see <a href="http://docs.jboss.org/hibernate/stable/core/reference/en/html/associations.html#assoc-bidirectional-join-121" target="_blank">One-to-One bidirectional association</a> on the Hibernate docs, specially the relation between the Person and PersonAddress tables).</li>
</ul></div><br />
The implemented Bad Data Model created two very serious structural problems. Let's try and understand how and why of these problems.<br />
<br />
<span class="Apple-style-span" style="font-size: x-large;">Temporal Causality no more</span><br />
<br />
The implemented Bad Data Model has one very important side-effect related with the record ordering.<br />
<br />
Given the structure of the Bad model, and because of performance reasons, the insert sequence looks like the following (this is just mockup code):<br />
<br />
<div class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">// create a transaction<br />
hibernateTrans.create();<br />
<br />
// insert into the ***SATELLITE*** tables<br />
hibernateTrans.save(extra);<br />
hibernateTrans.save(business);<br />
hibernateTrans.save(lowLevel);<br />
<br />
// set the satelite tables id in the main record<br />
trans.setIdExtra(extra.getId());<br />
trans.setIdBusiness(business.getId());<br />
trans.setIdLowLevel(lowLevel.getId());<br />
<br />
// save the main record<br />
hibernateTrans.save(trans);<br />
<br />
// commit the transaction<br />
hibernateTrans.commit();</div><br />
As you can see, the insert order is reversed compared to what common sense dictates: Transaction then satellites <i>versus</i> satellites then Transaction. This is because the Transaction table needs to store the references (IDs) to satellite tables and those are only available after the records are inserted. If the order is not reversed, then a later update must be executed on the already inserted Transaction record.<br />
<br />
As long as you execute in a <b>single threaded</b> environment, such as development, you can guarantee that for two consecutive transactions T1 and T2 (and the satellite tables) the invariant: "<i><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">T1.id < T2.id</span></i> <b>implies</b> <i><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">T1..Business.id < T2..Business.id"</span></i> holds true, the same can be said for the other satellite tables. The invariant provides temporal causality <b>across</b> tables of the Data Model. It does so because as IDs are monotonically increasing you can safely infer time causality on a table: lower id means the <i>insert happened before</i>.<br />
<br />
However, in a production environment, with the inserts running on <b>multiple threads</b>, that is no longer the case. To make it clear, imagine that transaction T1 is taken by thread T1 and that transaction T2 is taken by thread T2. The following image shows a possible sequence of execution:<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhsSF8Z5qOxDm2bOnvaybHHtf9FqcRnyV9DtZfFABVjYZEY8TV-1oUj6oE2Yoyh53at7hwEkO3f_5MmVaaxc1zQN-uFx4P_Zg4unH2p0fw98j9A103doHpsgKu03NWYBrVArbuf7g4X4k1n/s1600/ExecutionFlowSmall.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;" target="_blank"><img alt="Possible execution flow" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhsSF8Z5qOxDm2bOnvaybHHtf9FqcRnyV9DtZfFABVjYZEY8TV-1oUj6oE2Yoyh53at7hwEkO3f_5MmVaaxc1zQN-uFx4P_Zg4unH2p0fw98j9A103doHpsgKu03NWYBrVArbuf7g4X4k1n/s1600/ExecutionFlowSmall.png" /></a></div><br />
The green arrow marks the execution flow between the two threads (the horizontal lines are thread switches). You must remember that the selected ID for each inserted record is decided by the Database, based on the last used ID for each specific table.<br />
<br />
At the base of the image you can see the list of IDs assigned to each table: Transaction, Extra, Business, and LowLevel. As you can see, each list is a mixed set of values because the records are inserted in different order. Thus, the invariant does not hold.<br />
<br />
This seemingly unimportant fact has very deep implications for a Database server, because it is custom practice that records are physically stored on disk in Primary Key order. Thus, when you JOIN rows from the Transaction table with one of the satellites, records on both tables will not be on the same order forcing the Database to perform the extra step of ordering. This ordering step can be executed: 1) in memory (consuming a lot of resources); or 2) on the fly by making disk seeks for each matching record (not cache friendly).<br />
<br />
Now, contrast what happens with the Good Data Model. The sequence is to insert first the Transaction record and get the ID (primary key value). Then, the same ID is used to insert into the satellite tables. This holds the temporal causality invariant and forces the same order on all tables, thus enabling the database server to read records stored in the same physical order, in forward-only mode. This is very disk cache friendly and supports data read-ahead and data-prefetch algorithms.<br />
<br />
<i><b>Note</b>: for those of you that have the "concurrent and parallel programming" gene, perhaps i should make explicit the fact that loosing temporal causality is due to the lack of a "critical section" primitive when accessing the database. While the "get the next id" operation is atomic, you must realize that the Begin/Commit/Rollback transaction construct </i><u><i>only</i></u><i> guarantees that the sequence is executed or not at all, it says nothing about executing undisturbed. In fact, for many versions of MSSQL Server a Rollback doesn't undo the "get next id" associated with an Identity field, making it forever consumed. I think a rollback doesn't undo an Oracle Sequence either.</i><br />
<br />
<span class="Apple-style-span" style="font-size: x-large;">Query optimization is severely affected</span><br />
<br />
The second structural problem is perhaps more important, because it deeply affects query optimizer's ability to change query evaluation order. The best way to understand why is to think in terms of solving the following query:<br />
<pre class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">SELECT E.field1, B.field2
FROM Transaction T
INNER JOIN Extra E
ON T.idExtra = E.id AND /* conditions on Extra */
INNER JOIN Business B
ON T.idBusiness = B.id AND /* conditions on Business */</pre><br />
<i><b>Note</b>: the JOIN conditions for the <b>Good Model</b> will be <u>different</u>. For example, the Extra table JOIN will have T.id = E.id as condition instead of T.idExtra = E.id.</i><br />
<br />
Despite many differences between specific database products, we can think a generic database server will solve this by computing one partial temporary result (subquery) for each table (Transaction, Extra and Business). The server will then proceed to calculate the intersection between all the matching temporary results. In <a href="http://en.wikipedia.org/wiki/Relational_algebra">Relational Algebra</a> this matching is called a <a href="http://en.wikipedia.org/wiki/Relational_algebra#Natural_join_.28.E2.8B.88.29">Natural Join</a> and can be thought of as a <i>set intersection (<b>∩</b>)</i> or <i>and</i> operator. This last step can be represented then by the expression: <i>Transaction and Extra and Business</i>.<br />
<br />
A lot of the biggest optimizations happen when the server calculates this intersection. To understand how this happens and why, it is handy to think there are a set of functions that relate different tables of the Data Model. These functions have a single parameter (the Primary Key in the source table) and return the Primary Key of the associated record in another table. The next image represents these functions:<br />
<div class="separator" style="clear: both; text-align: center;"></div><br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrdcpOUwt_P-A9VSUNow9b7veQPC2R-SD2WsUECBwT9pOLHi8vzj09whaDLnbcBbVN1nXYEdD2Ugaw3BKF2PVagPfTLE8AHf_3C7FFqhtgrf9yd_rsXUIZkXi4KZzzjK-BbPzDhAJ6nmtG/s1600/Flow.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;" target="_blank"><img alt="List of Functions to relate tables in the Data Model" border="0" height="568" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrdcpOUwt_P-A9VSUNow9b7veQPC2R-SD2WsUECBwT9pOLHi8vzj09whaDLnbcBbVN1nXYEdD2Ugaw3BKF2PVagPfTLE8AHf_3C7FFqhtgrf9yd_rsXUIZkXi4KZzzjK-BbPzDhAJ6nmtG/s1600/Flow.PNG" width="536" /></a></div><br />
Note that there are a pair of functions between tables. For example, between Transaction and Extra table there are <i>E(t) ⇒ e</i> and the inverse <i>Ei(e) ⇒ t</i>. There are also relations between the satellite tables of the model. For clarity, the image shows only the relation functions between the Extra and Business table. Lets examine first the functions <i>E</i> and <i>Ei</i>.<br />
<br />
For the Bad Model, E(t) is very simple. Just return the field T.idExtra (see the JOIN condition of the example query). The inverse Ei(e) is more complex. You start with the primary key of Extra record (e) and then you must find the primary key of the associated Transaction record. To do that you must either: 1) read <u>every</u> Transaction record to find the one with <i>idExtra = e</i>, or 2) create an index on the idExtra field. The index is obviously faster, but it still requires some disk accesses.<br />
<br />
For the Good Model, the function E(t) is also simple. Just return the same t (remember JOIN conditions are different). The inverse Ei(e) is equally simple. Just return the same e. No extra indexes, no extra disk accesses. Just the <i>identity</i> function, as expressed by the JOIN conditions.<br />
<br />
Now just for fun, lets try to go from the Extra table to the Business table. The database will have to synthesize the functions <i>EtoB(e) ⇒ b</i> and <i>BtoE(b) ⇒ e</i>. EtoB(e) can be written as a function composition: <i>EtoB(e) = B(Ei(e))</i>. That is: given e, find the t that matches and then with that t, find b. Similar steps must be taken to create BtoE(b).<br />
<br />
For the Bad Model, Ei requires an index search while B just returns the idBusiness field of the specific Transaction record, but that means the database must <i><b>read</b></i> the record pointed to by e. For the Good Model, we know that <i>both</i> Ei and B are the <i>identity</i> function, so <i>B(Ei(e)) = Identity(Identity(e)) = Identity(e) = e</i>. No reads, no index searches. Again, just the <i>identity</i> function.<br />
<br />
Lets go back to our example and put all this in practice. Imagine the subqueries get executed and we get 500 results from table Extra and 1100 results from table Business, having a total of 1 million records in table Transactions (no conditions on Transactions on our example query). To compute <i>Transaction and Extra and Business</i>, our generic database will do one the following:<br />
<ul><li><b>Bad Model without indexes</b>: must go through each of the 1 million records of Transactions to be able to match the Extra and Business records, because the relations E(t) and B(t) go in one direction and the Ei(e) and Bi(b) require to scan the whole Transaction table (called a "tablescan"). To find a matching record by field idExtra or idBusiness you would have to perform a tablescan, so starting with table Transaction is the same. This means at least 1 million Transaction reads, plus the Extra and Business record sorting and matching.</li>
<li><b>Bad Model with indexes</b>: for each of the 500 results from table Extra, will have to do an index search to find the associated Transaction record to be able to match the corresponding Business record. Furthermore, the records on Business will not be in the same order that the records on Extra, so this requires a disk seek for each record. This means: 500 index searches on Transactions, 500 Transactions reads, 500 seeks on Business.</li>
<li><b>Good Model</b>: because of the <i>identity</i> function, the server knows that it can safely compute <i>(Transaction and (Extra and Business))</i> without altering results. That is: consider the intersection operation to be <i>associative</i>. Calculating the <i>Extra and Business</i> part involves a forward-only record match between 500 and 1100 records (same key on both tables, therefore same order). The records on table Transactions will have to be read because it's the table in the SELECT and not checking primary key existence on Transactions will break query semantics. This all means: forward only match of 500 and 1100 records (no internal sorting) and 500 forward only reads on Transactions.</li>
</ul><br />
The Bad Model breaks associativity because the relational function to go from one table to the next is not the <i>identity</i> function. The Bad Model with indexes is not as bad, but still causes a lot of disk seeks (the most expensive operation on disks). The Good Model preserves associativity and that enables the server to generate huge savings in disk seeks and reads. Database implementations have many other optimizations for <i>identity</i> key match, hence that kind of match should be used whenever possible.<br />
<br />
<span class="Apple-style-span" style="font-size: x-large;">The fix that is not</span><br />
<br />
The fix must be software only and it must involve both components of the application.<br />
<br />
First, the web service that inserts records must recreate the Object to Data mapping to use the Good data model. The relation fields idExtra, idBusiness and idLowLevel can be left blank (NULL value). Foreign Keys, if present, must be dropped from the Database.<br />
<br />
Second, the web application that queries records should be modified to be able to use the two data models, Good and Bad. The selection of the model to use is based on a system wide property that marks the moment in time where the Good data model began to be used. Because queries can span both models, a method of joining the results of querying both models in parallel must be implemented. The querying application is also able to do some basic reporting by returning aggregate counts. This reporting have to be changed to span both data models. Think about the complexity of calculating averages crossing different timespans.<br />
<br />
These changes are a full blown project on their own, but the biggest problem is that future changes and feature adds to the application will <b>duplicate</b> the testing effort as they must be tested on both models.<br />
<br />
A second alternative would be to create a new version of the application using the Good model and not to provide for model coexistence. With this approach, new installs will benefit, but your existing customers will be stuck with an unsupported version. This is not commercially viable, as this option implies to support two versions of the same application forever, <b>duplicating</b> development and testing.<br />
<br />
A third option was briefly considered but quickly discarded. It involved sending a T-101 to the past.<br />
<br />
The fix is that no fix would ever be applied. Data cannot be changed, Maintenance Costs cannot be increased. The problem is assumed to exist forever.<br />
<br />
<span class="Apple-style-span" style="font-size: x-large;">Conclusion</span><br />
<br />
There are a number of conclusions:<br />
<ul><li>Data Model and Object Model are two different things for a good reason. If you decide to force one onto the other, it better be a careful and informed decision.</li>
<li>Databases and ORMs are very complex tools; sometimes as complex as programming languages. It is your responsibility as a programmer to properly understand them. ORMs have been widely criticized or blindly used or both. SQL Databases have recently started to be very criticized too.</li>
<li>Local properties (Atomicity) do not imply global properties (Temporal Causality).</li>
<li>Once your application or product is deployed in a production environment, course corrections in the form of refactoring are not always possible. Many times logic is free to evolve while data is <b>anchored to past</b> versions of that logic.</li>
<li>In this particular case, the insert web service was programmed months before the query application, and by a different programmer. This translated into little initial engineering effort and set a bad course that is now impossible to correct.</li>
</ul><br />
This is long enough to be my fourth post.<br />
<br />
<b>PS</b>: many thanks to <a href="http://aleccolocco.blogspot.com/" target="_blank">Alejo Sanchez</a> for reviewing early drafts of this post and making so many suggestions to make it better.José Luis Campanellohttp://www.blogger.com/profile/00233954763631710324noreply@blogger.com0tag:blogger.com,1999:blog-2618135660518567589.post-26513030656821046592010-09-07T10:24:00.001-03:002010-09-07T10:25:22.746-03:00Exception madnessBefore <i>exceptions</i> became main-stream technology in programming languages (about 1.5 decades, i think), error control in programs was a delicate matter.<br />
<br />
The problem was due to a common design pitfall that plagued (and still plagues) many technologies. That pitfall is named: <b>in-band signaling</b>.<br />
<br />
<i><b>Note</b>: the name "in-band signaling" comes from the telecommunications industry (see </i><a href="http://en.wikipedia.org/wiki/In-band_signalling"><i>wikipedia entry</i></a><i>). The term comes from the fact that when you dial a number, the number itself is sent as sound over the line (you hear the tones, isn't it?). In-band signaling doesn't seem to be a problem, until you realize that other functions of the telecommunication network work the same way. Thus by knowing the right code, you can just dial it on your phone and lo and behold, the next call is not billed and things like that. That used to happen a lot in the 70', 80' and early 90' because of the modem cost of communications. I think it doesn't happen much today because once you pay for Internet access, the world is right at your fingertips.</i><br />
<br />
Just think about it. If you don't have the <i>exception infrastructure</i>, the exceptions must be then returned as a special form of result of your method or function.<br />
<br />
In languages like C, it is very common to see things like:<br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">FILE *file = NULL;</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">if ( (file = fopen("/some_file", "w")) != NULL) {</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> /* oops */</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> extern int errno;</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">};</span><br />
<br />
Notice that the function <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">fopen</span> is supposed to return a file handler (of type <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">FILE *</span>), but if an error occurs, then <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">NULL</span> is returned and you have to check the <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">errno</span> variable to see what went wrong.<br />
<br />
Do you see that the error is sent in-band with the result?<br />
<br />
This caused an awful lot of problems with many programs. Even to the point of using the most obnoxious pair of library calls in the C environment (be it setjmp and longjmp).<br />
<br />
What did this two calls provide? In plain english: the ability for a program to set(jmp) a recovery point were bad situations could be handled and the ability for that program to (long)jmp to that point when bad things happened. I have to say that you ought to be a terrorist to use these two function calls, so controlling for errors in-band infected regular programming as badly as a venereal decease.<br />
<br />
Perhaps you recognize that jumping back to a place where you know how to handle errors is just a tiny part of what the <i>exception infrastructure</i> provides in modern programming languages, but there is a lot more than just the jump in exception handling. That <i>lot more</i> is what made the pair setjmp/longjmp almost impossible to use properly. These jumps actually worked by just instantly moving the execution to a previous program scope, whereas exceptions destroy intermediate scopes (meaning: cleaning up stack objects and giving a chance to destroy manually allocated objects for languages without Garbage Collection).<br />
<br />
All of a sudden, with exceptions you can write much cleaner code. If, for instance, <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">fopen</span> would return an exception if the file cannot be opened, the following code will make a lot of sense:<br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">try {<br />
FILE *file = fopen("/some_file", "w");<br />
// do something with the file<br />
} catch (FileNotExistsException ex) {<br />
// handle error<br />
};</span><br />
<br />
However...<br />
<br />
<span class="Apple-style-span" style="font-size: x-large;">Bad things happen</span><br />
<br />
In our case, 3 bad things to be exact.<br />
<br />
The <b>first bad thing</b> is that the code to handle an exception gets separated from the code originating it. This actually increases complexity in our programs. Just consider our last example source code and imagine that the code implied by the line <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">// do something with the file</span> is actually 150 lines long (or even 15 lines long). Then, the line that generates the exception on the <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">catch</span> clause is not immediately obvious. In this case, the exception name will give a clue, but with more obscure exception names, the relation is not evident, just implicit and that augments complexity. You can of course put a specific try/catch pair for each call that can generate an exception, but that makes for bigger code. And think that we have not even considered the <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">finally</span> construct that is expected to be used to undo object creation side-effects. A comment stating the source instruction for each <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">catch</span> clause will help with this issue, but not solve it and is extra programmer effort.<br />
<br />
This leads to the <b>second bad thing</b>. Given that properly writing exception handling code is an extra effort, many programmers started to do one of two things: 1) put a generic <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">catch (Exception e)</span> clause, or 2) just not handling exceptions at all. The first case just looses the chance for fine grained error handling. A single exception is completely abortive of the function or method.<br />
<br />
The second case degenerates into the third bad thing, specially for web services or web applications.<br />
<br />
Enter the <b>third bad thing</b>. Considering bad thing number one and bad thing number two, proper exception handling is complex, so when we have a number of people writing code, the only we can count on is exceptions will not be properly handled. But don't despair. We can put a <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">catch (Exception e)</span> at the highest levels of our app and, at the worst, use an <i>all fucked up</i> generic error page. Even better, we can use the exception class to select a proper message to display, so it works like charm.<br />
<br />
Nothing particularly bad with that implementation, i even like it and use this idea, but only if it is not a replacement for proper error handling.<br />
<br />
The real (unintended) problem with this approach is that, when uncontrolled, it fosters...<br />
<br />
<span class="Apple-style-span" style="font-size: x-large;">Exception Madness</span><br />
<br />
Why is that? Well, when programmers can count on having a safety net below, they tend to get lazy (<b>we</b> tend to get lazy). Now, you can just throw exceptions for almost any code you don't want to write.<br />
<br />
For example, let's say you have an input field in a form that needs to be even (divisible by two). Some programmers (sadly not a few) will write a utility function or method called <i>even</i> that receives a generic object or integer and returns a boolean (true) if the value is even, <b>but</b> throw an exception if the value is odd. Of course, if you consider the method name there is no good reason for it to throw anything other that <i>InvalidParameterType</i> if it accepts general objects as input; but if you are going to throw your stomach contents on to your caller, at least have the decency to call the method something like <i>ThrowsIfNotOdd</i>.<br />
<br />
These kind of situations are doubly perverse. First because they abuse the last resort exception handling and second because they do not put business or application logic in the right place (if you are going to abuse, then the caller should throw, not the utility function).<br />
<br />
<span class="Apple-style-span" style="font-size: x-large;">Conclusion</span><br />
<br />
<i>Exceptions</i> are actually a step ahead in error handling, a step i am glad was introduced to many programming languages; but i think the problem is that once again, we got the silver <i>bullet syndrome</i> with them. Exceptions are an excellent tool, but only to the extent they are used properly, and they have some side effects too that you should be aware of.<br />
<br />
Enough for a very very very late third post. Sorry about that.José Luis Campanellohttp://www.blogger.com/profile/00233954763631710324noreply@blogger.com4tag:blogger.com,1999:blog-2618135660518567589.post-42925238701764172012009-12-24T19:43:00.003-03:002009-12-26T17:59:17.116-03:00The flavors of Java concurrency controlThis second post took longer than i expected. I was involved in a witch hunt in my day job for a few weeks and then everything else started piling up. The witch hunt got me into reviewing a lot of Java code running inside an <a href="http://en.wikipedia.org/wiki/Enterprise_service_bus">ESB</a> to find any kind of performance hog i could spot with the naked eye. Yes, i know there are profiles, but good luck trying to run one of those things in a Bank's production environment.<br />
<br />
During the review, i found a few possible enhancements, the most important one of those is what started this post.<br />
<br />
I will discuss two methods of concurrency control available in Java 1.5, the <b>synchronized</b> modifier and the classes in the <b>java.util.concurrent.atomic</b> package, then some conclusions. I will not use java specific objects when trying to explain some language-neutral feature (a comment with the word "LN" will be added to such lines).<br />
<br />
<span style="font-size: x-large;">The synchronized modifier</span><br />
<br />
This modifier provides a (no-brainer) way to implement mutual exclusion. You prepend the modifier to a method and then that method becomes guarded by a mutex. What comes to our minds when we read a sentence like the one before is that, for the following code:<br />
<br />
<span style="font-family: 'Courier New', Courier, monospace;">int someValue = 0;</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"></span><br />
<span style="font-family: 'Courier New', Courier, monospace;">synchronized int getNextValue() {</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> return(someValue++);</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">}</span><br />
<br />
the compiler will generate code looking like:<br />
<br />
<span style="font-family: 'Courier New', Courier, monospace;">int someValue = 0;</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">Mutex getNextValue_mutex = new Mutex();</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"></span><br />
<span style="font-family: 'Courier New', Courier, monospace;">int getNextValue() {</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> getNextValue_mutex.getLock(); // LN</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> int tempValue = someValue++;</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> getNextValue_mutex.releaseLock(); // LN</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> return(tempValue);</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">};</span><br />
<br />
It is reasonable to expect that the Mutex.getLock allows only one thread to get the lock, while queueing other threads in a FIFO way; while the Mutex.releaseLock takes the first queued thread (if any) and schedules it for execution.<br />
<br />
And that is what the Java synchronized modifier does. Well, not quite... it turns out that this modifier does a little more than just guard the method with a mutex (see <a href="http://www.ibm.com/developerworks/java/library/j-threads1.html">IBM's article Threading lightly, Part 1: Synchronization is not the enemy</a>.)<br />
<br />
The truth is that Java's synchronized modifier does flush the processor's data cache on entry and commits it to memory on exit. So the real code executed will look more like:<br />
<br />
<span style="font-family: 'Courier New', Courier, monospace;">int someValue = 0;</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">Mutex getNextValue_mutex = new Mutex();</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"></span><br />
<span style="font-family: 'Courier New', Courier, monospace;">int getNextValue() {</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> getNextValue_mutex.getLock(); // LN</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> Native.processorDataCacheFlush(); // LN</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> int tempValue = someValue++;</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> Native.processorDataCacheCommit(); // LN</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> getNextValue_mutex.releaseLock(); // LN</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> return(tempValue);</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">};</span><br />
<br />
<i><b>Sidenote</b>: the Native class is there representing some low-level methods to handle the processor cache (written either in C or Assembly and completely platform dependent). Also, the whole cache needs not to be discarded, but figuring out what can be kept is not an easy problem to solve.</i><br />
<br />
Two things to notice here:<br />
<br />
<ul><li>If you come to think about this, the synchronized modifier will work only if the processor's data cache is handled this way, and</li>
<li>I suspect that Microsoft's .Net synchronized modifier works the same way.</li>
</ul><br />
The data cache must be flushed on entry because you really don't know the status of precached data the processor might have. It is possible that on entry to the method, the processor's cached data was fetched before another processor executed the same method, rendering the cached data unusable.<br />
<br />
So yes, you must do that nasty thing to the processor's cache. Nasty being the right word here. Just consider that Intel, AMD and every other manufacturer devotes a very large number of transistors on every chip to data caches and predictive data pre-fetch.<br />
<br />
There is a very important reason for this expenditure in transistors. Your Gb sized main memory is slow compared to the motherboard's bus frequency which is in turn very slow compared to the CPU's clock frequency. For example, <a href="http://en.wikipedia.org/wiki/DDR2_SDRAM">DDR2 SDRAM</a> has it's own clock running at half the speed of the motherboard's bus clock frequency which runs a fraction of speed of CPU's clock frequency. Some set of common values are: 266 Mhz / 533 Mhz / 2.1 Ghz (memory / bus / CPU).<br />
<br />
To keep the CPU busy, the pre-fetch logic scans ahead the instruction stream to guarantee that the data stored in memory is at hand when the processor needs it. Consider that flushing the processor's data cache throws away all the good work the pre-fetch logic has been doing, meaning that the CPU has to sit idle until the pre-fetch has a chance to start refilling the caches.<br />
<br />
This has a terrible effect on performance. Assuming that the cache flush affects only the entries of your program your other threads suffer from it. Imagine what the effect will be if the flush is not selective and discards also cached data from the operating system, services, etc.<br />
<br />
There are two more things to pounder. The first one is that this cache flushing and committing should only matter in multiprocessor systems (like your servers.) That is, up until now, in a single multi-core processor system all cores in the processor share the same cache, yet i think that abundance of transistors is or will be changing that soon. I have not been able to confirm what the JVM does in these cases (i.e: if it really optimizes the single processor case by not doing the cache dumping.) But i don't think that's very important... How many people buy a single processor server these days?<br />
<br />
The second thing to consider is that Intel and AMD are both talking about cache coherency and invalidation protocols for inter-processor communication. The idea is that if you have one processor changing a memory position, then it will broadcast that fact to other processors, so rather than invalidating their whole caches, the other processors can just drop the outdated cache entry. I think there are already some Intel processors able to do that, but i'm not sure of this.<br />
<br />
To recap, the synchronized modifier uses a mutex that provides for MUTual EXclussion and guarantees that threads waiting for the lock are given the lock in the order they requested it. This also requires processor's data cache flushes and commits and, depending on the platform you are running, it also requires expensive context switches to kernel mode (to operate on the mutex itself.) Threads are de-scheduled when they can't immediately get the lock so they don't consume CPU if not able to proceed.<br />
<br />
<span style="font-size: x-large;">The atomic package</span><br />
<br />
The atomic package (java.util.concurrent.atomic) was added on Java 5. This package provides a set of thread-safe read/write atomic scalar values equivalents (boolean, integer, long, references and array of these). The big thing behind these replacements is that they are lock free (no mutexes) and thread-safe.<br />
<br />
The Synchronized modifier example we've seen in the begin of the previous section could be written as:<br />
<br />
<span style="font-family: 'Courier New', Courier, monospace;">AtomicInteger someValue = new AtomicInteger();</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"><br />
</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">int getNextValue() {</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> return(someValue.getAndIncrement());</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">}</span><br />
<br />
Notice that the synchronized word disappeared and, being thread safe, no thread will get a partially updated value.<br />
<br />
How is this possible? Thanks to a technique called Spin Lock. A Spin Lock is a method that, AFAIK, was devised first for intra-operating system synchronization in the face of multiple CPUs. If you had something resembling an Operating Systems course, you'll remember the Test and Set synchronization method (if you don't, then you can read about it on <a href="http://en.wikipedia.org/wiki/Test_and_set">Wikipedia</a>.) A Spin Lock is a loop around a Test and Set operation. This will make a program to keep looping (consuming CPU) until the lock is obtained. If you want to know more about Spin Locks (and many other synchronization options), check reference [1].<br />
<br />
On Intel architectures, there is a single processor instruction that exchanges two data values. One variant of this instruction (taking a register and a memory address) can be used to implement an atomic exchange or, put in other words, a Test and Set. The instruction is atomic because it automatically places a low level lock on the motherboard's bus so that no other component can access the system while the lock is held (and as memory is connected to the bus, only the lock holder can use it).<br />
<br />
<i><b>Sidenote</b>: this instruction can be traced on Intel architectures back to the 8086 and 8088 processors, where it required a special prefix to place a lock on the bus. In case you are thinking about it, the answer is: yes! As far as 1978, Intel processors have been multiprocessing-ready.</i><br />
<br />
Now, what about the processor's data cache and the exchange instruction? Well, as you might have guessed, for this to work, the Test And Set must be cache ignorant. That is, this particular instruction causes the pre-fetch logic to ignore it, because the pre-fetched data could be altered after fetch but before use. Also, it is slow compared to other instructions, because it has to access memory right on the spot. This "slow" means that the whole system is penalized by a few bus clock cycles. The reality is of course a bit more complicated, because of dual gate memories, multiple buses, etc. But you should get the picture.<br />
<br />
There is a sister package called java.util.concurrent, that provides some data structures (HashMap, Array, ThreadPools, etc) that use the Spin Lock mechanism to provide concurrent data reads and writes.<br />
<br />
To recap, the Spin Lock method is implemented always at the process level (does not require the operating system's help.) It does not require the processor's data cache to be flushed, but it slows down the whole system (by a few clock cycles because of cache bypassing.) Threads are not de-scheduled when they are unable to get the lock. They are allowed to keep looping (consuming CPU) until they get the lock or the assigned CPU slot is fully consumed. This makes impossible to guarantee that threads get the lock in any particular order. It also means that the things that you can guard by use of a Spin Lock should be really fast and non-blocking (ex: no file or network I/O).<br />
<br />
<span style="font-size: x-large;">Why are there two methods?</span><br />
<br />
Well, that's easy: these are two different tools that are used for two similar yet different tasks, in different scenarios. You just have to know when to use which one.<br />
<br />
The definitive answer is of course dependent on the full description of the problem you are solving, but the following list of questions is more or less my recipe to decide:<br />
<br />
<ul><li>Do you need a strict FIFO access to the shared resource? If yes, you must use the synchronized method. Example: if e-bay receives two bids for the same price, they want the first incoming one to win.</li>
<li>Does the code to be executed is fast and small and has no I/O or other blocking operation? If yes, then you can go with the Spin Lock. Remember that the Spin Lock keeps consuming CPU while the lock is not acquired.</li>
<li>Are you optimistic or pessimistic about the amount of contention on the resource? If you are pessimistic, then go with the synchronized method. If you are optimistic go with the Spin Lock variant. Here optimistic means that there will be low contention on the resource, so the busy loop of the Spin Lock will outperform the context switch (for mutex) and thread de-schedule of the synchronized case.</li>
</ul><br />
Besides that, i think it's safe to assume that if you need a data structure that doesn't change too often and you want maximum possible concurrency, the concurrent data structures provided by java.util.concurrent are in general a good option, but...<br />
<br />
In a congested (CPU) system, the Spin Lock's "busy loop" could make the congestion even worse by wasting more CPU. Ideally a Spin Lock should try for some amount of time and then yield the CPU to allow for the lock holder to process and the lock to be released, but this can cause the thread to be delayed beyond what is acceptable under high CPU loads.<br />
<br />
<span style="font-size: x-large;">Conclusion</span><br />
<br />
You should keep these two concurrency control options at hand in your tool-belt, but you should always start by using the most important of the tools you have at your disposal (AKA your brain) and don't get fooled.<br />
<br />
One last example. A few months back, there was a different system suffering performance issues. One of the problems turned out to be that a one way (audit and trace) message was put on a queue to be processed in background. The decision to do this was sound, because the extra processing to be done could be made asynchronously in a different machine, making the foreground task faster, without sacrificing the audit functionality.<br />
<br />
How was that a performance problem? Well, queues are complex data structures and as such, require the put and get operations to be synchronous (locking the queue.) This case was even worse, as the queue was part of a clustered ESB having multiple producers and consumers, on different machines.<br />
<br />
While the system had an average load, everything went peachy, but when the load started to peak, the overhead caused by locks on the queue put/get operations began to be noticeable, up to the point where performance was hit.<br />
<br />
Synchronization is all around you, even if you don't see it.<br />
<br />
Enough for a second post.<br />
<br />
<b>UPDATE</b>: many thanks to <a href="http://aleccolocco.blogspot.com/">Alejo Sanchez</a> for reviewing early drafts of this post.<br />
<br />
<span style="font-size: x-large;">References</span><br />
<br />
[1] John M. Mellor-Crummey, Michael L. Scott, <a href="http://doi.acm.org/10.1145/103727.103729">Algorithms for scalable synchronization on shared-memory multiprocessors</a>, ACM Transactions on Computer Systems (TOCS) Volume 9 , Issue 1 (Feb. 1991) Pages: 21 - 65. You can download it for free from <a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.81.9990&rep=rep1&type=pdf">SiteSeer at Penn State University</a>.José Luis Campanellohttp://www.blogger.com/profile/00233954763631710324noreply@blogger.com0tag:blogger.com,1999:blog-2618135660518567589.post-44662497945591114052009-11-08T13:07:00.000-03:002009-12-26T18:01:30.697-03:00printf("Hello World!\n");<span style="font-family: 'Lucida Grande'; font-size: small;"><span class="Apple-style-span" style="font-size: 11px;"><span style="font-family: Times;"><span class="Apple-style-span" style="font-size: medium;"><span style="font-family: 'Lucida Grande'; font-size: small;"><span class="Apple-style-span" style="font-size: 11px;"><span style="font-family: Times;"><span class="Apple-style-span" style="font-size: medium;">Not much else to say, isn't it?</span></span></span></span></span></span></span></span><br />
<span style="font-family: 'Lucida Grande'; font-size: small;"><span class="Apple-style-span" style="font-size: 11px;"><span style="font-family: Times;"><span class="Apple-style-span" style="font-size: medium;"><span style="font-family: 'Lucida Grande'; font-size: small;"><span class="Apple-style-span" style="font-size: 11px;"><span style="font-family: Times;"><span class="Apple-style-span" style="font-size: medium;"><br />
</span></span></span></span></span></span></span></span><br />
<span style="font-family: 'Lucida Grande'; font-size: small;"><span class="Apple-style-span" style="font-size: 11px;"><span style="font-family: Times;"><span class="Apple-style-span" style="font-size: medium;"><span style="font-family: 'Lucida Grande'; font-size: small;"><span class="Apple-style-span" style="font-size: 11px;"><span style="font-family: Times;"><span class="Apple-style-span" style="font-size: medium;">I mean, if you are a programmer, then you get the thing. If not, you are probably in the wrong place.</span></span></span></span></span></span></span></span><br />
<span style="font-family: 'Lucida Grande'; font-size: small;"><span class="Apple-style-span" style="font-size: 11px;"><span style="font-family: Times;"><span class="Apple-style-span" style="font-size: medium;"><span style="font-family: 'Lucida Grande'; font-size: small;"><span class="Apple-style-span" style="font-size: 11px;"><span style="font-family: Times;"><span class="Apple-style-span" style="font-size: medium;"><br />
</span></span></span></span></span></span></span></span><br />
<span style="font-family: 'Lucida Grande'; font-size: small;"><span class="Apple-style-span" style="font-size: 11px;"><span style="font-family: Times;"><span class="Apple-style-span" style="font-size: medium;"><span style="font-family: 'Lucida Grande'; font-size: small;"><span class="Apple-style-span" style="font-size: 11px;"><span style="font-family: Times;"><span class="Apple-style-span" style="font-size: medium;">Enough for a first post.</span></span></span></span></span></span></span></span>José Luis Campanellohttp://www.blogger.com/profile/00233954763631710324noreply@blogger.com0