JavaandPythonCoGKits
KeithR.Jacksonkrjackson@lbl.govGregorvonLaszewskigregor@mcs.anl.gov
1 Objectives •WhataretheCoGKits,andwhydoIwanttousethem?
•HowdoIusetheCoGKits?
•HowdoIdevelopmyponentswith theCoGKits?
•Future
2 MotivationforCoGKits •Problem –ManyapplicationdevelopersdesiretoprogramtheGridinframeworksfamiliartothemtoforexampleenablerapidprototyping. •Solution –TheCoGKitprojectintegratesGridsoftwarebasedontheGlobusToolkitandmodityframeworksuchasJavaorPython. •EasierdevelopmentofadvancedGridservices•Easierandmorerapidapplicationdevelopment•EasierdeploymentofGridservices•Codereuseanduseponentrepositories•UseofWebservicesaspartoftheGrids•WidespreaduseoftheGrid
3 CoGsaremore… •CoGsaremorethanjustaninterfacetotheGlobusToolkit •CoGsallowGridprogrammerstousetheCommodityTechnologiesANDtheGridsadvantages –Example:EventandexceptionmodelofJava–Example:SWIGwrappersinPythonforrapid prototypingandlegacycodesupport. •Thus,CoGsarenotjustanAPIbutprovideesstotheCommodityFramework
4 ReasonsforusingtheJavaCoGKit •WhyusetheJavaCoGKit?
–keepingupwithpatchesandprotocolchangesinGlobushasbeendifficultinthepast –TheJavaCoGKithassofarprovidedmunitywitha“service”tohidetheburdenofchangingyourcode. •AskforatestimonybytheXCATgroupatIndianaUniversity. –ManyusersneedonlythefeaturesprovidedintheJavaCoGKit–Little/nochangesweresofarinvolvedtoswitchbetweendifferent versionsoftheGlobusToolkit–Mostlyclientsideprogramming–JavaPlatform: •HighlevelframeworkbettersuitedforGridprogrammingthanC •WhynotusetheJavaCoGKit –ItdoesnotprovideallthelatestfeaturesoftheGlobusToolkit.–ItusesJava,andsomeinmunity…;-)
5 ReasonsforUsingthePythonCoGKit •WhyusethePythonCoGKit?
–ProvidesafullinterfacetotheGlobusToolkit–Little/nochangesareinvolvedinswitchingbetween differentversionsoftheGlobusToolkit–HighlevellanguageallowsforeasierGridprogramming •SupportsrapidprototypingofGridservices/applications –ManyautomatedtoolsexistforexposinglegacyC/C++orFortrancodesasPythonobjects •WhynotusethePythonCoGKit –Smallperformancepenaltyforusinganyinterpretedlanguage •MinimizedbecausethePythonCoGKitisathinwrapperoverthenativeCcode. –Nostatictypechecking–ItusesPython,andsomeinmunity…;-)
6 Motivation:JavaandPythonCoGKits •UseandleverageexistingtechnologiesforGridprogramming-ThecapabilitiesoftheframeworkontowhichGridServicesaremappedcanbeexploited:Objects,Events,Exceptions,JNDI,...-Objectslikejobs/taskscanbedefined.-XMLsupportisprovided.-GUI's,....,IDE'scanbeused(Forte,BOAConstructor…) •Maximizesoftwareflexibility,extensibility,andreusability•Providefoundationsforapplicationdeveloperteamsthatare familiartodevelopapplicationsinthisframework-Reducedevelopmentandmaintenancecost •Useasglueformanytechnologies•Pythoniswellsuitedtotyingtogethermanydifferentlanguages/technologies
7 WhatistheJavaCoGKit?
•TheJavaCoGKitprovidesamappingbetweenJavaandtheGlobusToolkit.ItextendstheuseofGlobusbyenablingtoessadvancedJavafeaturessuchaseventsandobjectsforGridprogramming. •TheJavaCoGKitisimplementedinpureJava.ItspeakstheGridprotocols. •ItisnotawrapperoftheCGlobusToolkit•Thisallowsintegrationwithinapplets.•Mostlyclientsidesupport
8 WhatisthePythonCoGKit?
•SimilarlythePythonCoGKitprovidesamappingbetweenPythonandtheGlobusToolkit.ItextendstheuseofGlobusbyenablingtoessadvancedPythonfeaturessuchasexceptionsandobjectsforGridprogramming. •ThePythonCoGKitisimplementedasaseriesofPythonextensionmodulesthatwraptheGlobusCcode. •UsesSWIG()tohelpgeneratetheinterfaces.
9 Status:JavaCoGKit •Modifiedcoreponents(Protocols) •Basicservicesareprovidedessing: –Security (GSI) –Remotejobsubmissionandmonitoring(GRAM) –RemoteDataess (GridFTP) –InformationServiceess (MDS) –Certificatestore (myProxy) •Current100%clientponentsincludes •ReusableGridponents •AvarietyofGridInterfaces 10 Status:PythonCoGKit •Basicservicesareprovidedessing: –Security (security) –Remotejobsubmissionandmonitoring(gramClient) –SecureworkIO (io) –Protocolindependentdatatransfers (gassCopy) –HighperformanceGridFTPtransfers (ftpClient) –SupportforbuildingGridFTPservers (ftpControl) –RemotefileIO (gassFile) –InformationServicesess •Highlevelservicesforeasierusage •Taskbasedservicestomonusagepatterns 11 Communication •WebPage –/cog–www-itg.lbl.gov/gtg/projects/pyGlobus/ •Bugs –Java: •/cog/contact/bugs/IsmaintainedbytheGlobusProjectbutdoesnotcontainsomeofthements •http://arbat.mcs.anl.gov:8080/bugzilla/Isouroldbugzilla –Python: •http://www-itg.lbl.gov/bugzilla/ •Download –/cog–/cog/java/–www-itg.lbl.gov/gtg/projects/download/download_info.html 12 Versions •JavaVersions –0.9.13isapatiblewithGlobus2.0 •wemendusingGlobus2.0ifyouliketousethisrelease –1.1apatiblewithGlobusToolkit2.2andinfutureGlobusToolkit3.0,e.g.muchofitmaybedistributedaspartofGlobusToolkit3.0 •WemendusingthisversionforGlobus2.2•Stillanalpharelease •PythonVersions –0.9.8 •Twoversions,oneforGT2.2andoneforGT2.0 13 Download&Compile 14 JavaCoGSetupOptions •DownloadSourceorbinary –/cog •Binary –ReadtheReadme–setup–changetheenvironmentvariables •Source –Readthereadme–setup–changetheenvironmentvariables •Manualwillbeavailableshortly 15 PythonCoGKitSetupOptions •Downloadsource –SetEnvironmentvariables–Compile–Install •EasytocreateRPM’sforLinuxsystems•Binaryingforwin32 –AssoonasGlobusofficiallyreleasesthewin32port •Canbuildbinarypackagesforanyplatform•UsesthestandardPythondistutilmodule 16 Requirements •JavaCoG –JDK1.3.1or1.4.1 • –ant •/ant/ •PythonCoG –Python2.0+ • –GlobusToolkitInstallation–GPTInstallation 17 JavaCoGCompilation •cdwork •wecreateabuild •cdogce•antdist•cd..•
directoryinwhichthejarfilesandshellscriptsarelocatedthatallow
–(a)programmingand
•work–ogce
–(b)useofclienttoolsfrommandline
–jglobus
–build
•cog-1.1a
FAQ.TXT
bin//herearemands
lib//herearethejars
etc
18
PythonCoGKitCompilation
•EnsurethatGPT_LOCATIONandGLOBUS_LOCATIONaresetappropriately
•cdpyGlobus-{Version}•pythonsetup.pybuild
–--prefix=/path/to/installation/directory–Willonlybuildthosepackagesthatyouhave
Globusinstallsof
•pythonsetup.pyinstall
–Toinstallinthesite-extensionsdirectoryrequiresrootprivilege
19
ObtainKeys
•ObtainGlobuscredentials
–Userprivatekey–Usercertificate–TrustedCAcertificates–Takealookat/security
•ObtainacertificatefromyouranizationCA•youcaninstallasimpleCAonyourmachinetocreateyour
owncertificates.•youcouldinstallpartoftheGlobustoolkitthatwouldallow
youtorunglobus-cert-request
•Placecredentialsmonplace
–Usershomedirectoryin.globusdirectory
20
SetupoftheJavaCoGKit
21
Setuphastwopurposes
•Enableessmandlines(optional)
–pathandotherenvironmentvariables
•Enableesstocertificates(mandatory)
–convenientponent
22
Usingmandlinetools(optional)
•addwork/build/cog1.0a/bin/toyourpath•Unix
–exportPATH=$PATH:/home/gregor/work/build/cog1.0a/bin–exportJAVA_INSTALL_PATH=/home/gregor/work/build/cog1.0a–shouldbe:;-)JAVA_COG_HOME=/home/gregor/work/build/cog1.0a
•Windows
–Registry
•ControlPanel->System->Advanced->EnvironmentVariables•NewVariableJAVA_COG_HOME•EditthePathVariable
23
SetupComponent(mandatory)
24
SettinguptheJavaCOGKit
•Thisinvolveswritingacog.propertiesfiletoyouruser.globusdirectorywhichcontainsinformationregardingthefollowing:
–Thelocationofyourusercertificatefilethatidentifiesyouasatrusteduserandisusuallynamedusercert.pem
–Thelocationofyourprivatekeyfilethatusuallytakesthenameuserkey.pemandisusedtoencryptinformationyousendout,sothatrecipientswithapublickeycouldverifyyouridentity
–ThelocationoftheCertificateAuthority(CA)file.TheCAisathirdpartythatisusedtocertifythelinkbetweenthepublickeyandthesubjectinthecertificate.
–Thelocationoftheproxyfilethatisusedtotemporarilyidentifyyouonthegrid.Thefileusuallytakesthenamex509up_u_usernameandcanoftenbefoundinyourtemporarydirectory.
–ThesystemIPaddressandthedatealsoneedstobechecked.
Itisadvisedthatthecertificatesalsobestoredinthedefault.globusdirectory.
25
Examplecog.propertiesfile
•Youcancustomizethecog.propertiesfilesbyhand.
–seetheFAQ.TXTinthebindirectory
•relevantCode
–work/jglobus//mon/CoGProperties.java–work/ogce//globus/ogce/gui/cogsetup
•Commonvaluessetincog.properties–usercert=/home/globoid/.globus/usercert.pem–userkey=/home/globoid/.globus/userkey.pem–proxy=/tmp/x509up_u999–cacert=/etc/grid-security/certificates/42864e48.0,/etc/grid-security/certificates/5aba75cb.0–ip=192.123.123.1
26
ponents
OneexampleofauserinterfacetotheGrid
27
JavaBeans
•LocalSystemFileBrowsing
LocalTreeFrameBeanprovidesthecapabilitytobrowsethefilespresentinthelocalsystem.
•RemoteSystemfileBrowsing
RemoteTreeFrameBeanisthebasicbean,whichprovidesgenericremotefilebrowsingfunctionalities.CurrentlyweprovideGridClientandFtpClientbeansthatutilizetheRemoteTreeFrametoprovideimplementationforftpandgridftpserversrespectively.
•FileTransfer
FileTranferBeanprovidesthefiletransfercapabilities.
Available:.globus.ogce.beans.filetransfer.*
28
FileTransferComponent
29
ShellComponent
30
ShellComponentmands)
•Help
–Printsoutallmandssupportedbytheshell–helpmand>givesspecificsyntaxhelp
•Basicmands
–mkdir,cd,ls,pwd,rm,rmdir,cp,cls,exit…
•FileTransferConsolestoperformallfiletransferoperations.
–ftptoopenaconnectiontoFTPsite–gridftptoopenaconnectiontogridFTPsite
31
ShellComponentmands)
•mands
–globusrun
runjobsonremotemachine
–globus-url-copycopyfiles
–globus-jobsetsubmitsetofjobswithdependenciestoaremotemachine
•BackgroundThreadoption(&)mands
–ps
viewalltherunningprocesses
–resume
resumeaprocess
–suspend
suspendaprocess
–kill
killaprocessthread
–result
seetheoutputoftheprocess
•Batchfiles
–ant–fgrant.xml–runfilename
32
VisualJobSubmission
Authentication(EnterPassphrase)
SelectMachine
SpecifyJob
Observeoutput
Hello123Bang
World
Atesttostderr321Bang
RuntheJob
33
GridInformation/LDAPBrowser
•Download
–http://www-unix.mcs.anl.gov/~gawor/ldap/
•Separatelydistributed
–Historicalreasons,oneofthefirstJavaCoGponents–thousandsofusers
•alsooutsideofmunity
34
•Familiarlookandfeel•prototypeforwhatyou
cando
35
Grid
Loginwithmachinesandjobs
ObservethejobsSubmittedtoamachine
SpecifytheJob
ObservetheOutput
36
WorkflowComponent
•underdevelopment
37
JavaCommandLineTools
38
JavaCommandLineTools
•Security
–grid-proxy-init
authenticatetogrid
–grid-cert-info
givescertinformation
–grid-proxy-destroydestroytheproxy
–myproxy
startsamyproxy-server
•FileTransfer
–globus-url-copy
simpleURL-to-URLcopy
•Jobexecution
–globusrun
runbothbatchandinteractivejobs
•PersonalGatekeeper
–globus-personal-gatekeeperrunasimpleserver
•GassServerfortranferingfilesforexecution
–globus-gass-server
startsthegassserver
–globus-gass-server-shutdownshutsdowntheserver
39
SetupforRunningTools
•COG_INSTALL_PATHandPATHarethevariablesyouneedtosetupinordertorunthetools.
•Procedure:
–BuildandinstalltheJavaCoGKitpackage.–SettheCOG_INSTALL_PATHtotheplacewherethecogis
built.–Includethecogbindirectoryintothepath.
•Example(sh)
–setCOG_INSTALL_PATH=C:\build\cog-1.0a–setPATH=%COG_INSTALL_PATH%\bin
40
HowtoProgramwiththeJavaandPythonCoGKits
41
Examples
•Javaexamples:
–severalbasicexamplesarein
•/globus/examples
–severaladvancedexamplesareintheappropriatesubdirectory
•e.g.securityexamples
•Pythonexamples:
–Basicexamplesarein
•pyGlobus/examples
–Testdirectoriescontaintheunittestcodethatprovidemoreadvancedexamples
42
ResourceSpecificationLanguage
•RSLismoninterchangelanguagetodescriberesources,irrespectiveoftheschedulerorbatchsystemused.
Supportsattributeslike
•executable,workingdirectory,argumentslist
•stdin,stdout,stderr(localfilesorGass/FTPURLs)
•min/maxmemory,maxcputime,no.ofprocesses
•Etc.
•.globus.rsl.RslAttributes
convenientmethodsforRSLexpressioncreationandmanipulation
RslAttributesrsl=newRslAttributes();
rsl.add(“executable”,“/bin/ls”);
StringrslStr=rsl.toRSL();
//returnedRSL:&(executable=/bin/ls)
rsl.remove(“executable”);
43
JavaJobSubmission
•Checkwhetheryoucansubmitajobtoaparticulargatekeeper.
Gram.ping(proxy,“hot.mcs.anl.gov”);
•Createajob
GramJobjob=newGramJob(proxy,rsl.toRSL());
•Addastatuschangelistener
classGramJobListenerImplimplementsGramJobListener{publicvoidstatusChanged(GramJobjob){Stringstatus=job.getStatusAsString();}
job.addListener(newGramJobListenerImpl());
•SubmitthejobtoaGRAMresourcemanager
job.request(“hot.mcs.anl.gov”);//defaultIANAport2119
•Cancelthejob,ifneedbe.
job.cancel()
44
JavaOutputRedirection
•CreateaGASSservertoreceiveoutput/error
ForaGASSserveronclientmachine:GassServergass=newGassServer(proxy,port);
AGASSservercanalsobestartedonaremotemachinethroughthegatekeeper.
•SettheGASSURLasstdout/stderrparameterinjobRSL.Thiswillstreamthejoboutput/errortotheGASSserver.
rsl.add(“stdout”,gass.getURL()+"/dev/stdout“);rsl.add(“stderr”,gass.getURL()+"/dev/stderr“);
•Redirecttheoutput/errorfromGASSservertoaJobOutputListener.
classJobOutputListenerImplimplementsJobOutputListener{publicvoidoutputChanged(Strings){/*processtheoutputhere*/}publicvoidoutputClosed(){}};
JobOutputListenerImploutListener=newJobOutputListenerImpl();JobOutputStreamoutStream=newJobOutputStream(outListener);gass.registerJobOutputStream(“out”,outStream)gass.registerJobOutputStream(“err”,outStream)
•Output/errorsfromdifferentjobscanbehandledbyasingleGASSserver.
45
PythonJobSubmissionExample
•Creatingajob.
try:gramClient=GramClient.GramClient()callbackContact=gramClient.set_callback(func,condV)jobContact=gramClient.submit_request(“clipper.lbl.gov”,
“&(executable=/bin/sleep)(argument=15)”,GramClient.JOB_STATE_ALL)
exceptGramClient.GramClientException,ex:printex.msg
•Callbackforstatechanges.
deffunc(cv,contact,state,error):ifstate==GramClient.JOB_STATE_PENDING:print"Jobispending"elifstate==GramClient.JOB_STATE_ACTIVE:print"Jobisactive"
46
RedirectingstdoutwithgramClient
frompyGlobusimportgassServerEZopts=gassServerEZ.STDOUT_ENABLEserver=gassServerEZ.GassServerEZ(opts)url=server.getURL()rsl="&(executable=/bin/sleep)(arguments=15)
(stdout=%s/dev/stdout)“%url
47
Compare:CJobSubmissionExample
callback_func(void*user_arg,char*job_contact,
intstate,interrorcode)
{
globus_i_globusrun_gram_monitor_t*monitor;
monitor=(globus_i_globusrun_gram_monitor_t*)user_arg;
globus_mutex_lock(&monitor->mutex);
monitor->job_state=state;
switch(state)
{
caseGLOBUS_GRAM_PROTOCOL_JOB_STATE_PENDING:
{
globus_i_globusrun_gram_monitor_t*monitor;
monitor=(globus_i_globusrun_gram_monitor_t*)user_arg;
globus_mutex_lock(&monitor->mutex);
monitor->job_state=state;
switch(state)
{
caseGLOBUS_GRAM_PROTOCOL_JOB_STATE_FAILED:
if(monitor->verbose)
{globus_libc_printf("GLOBUS_GRAM_PROTOCOL_JOB_STATE_FAILED\n");
}
monitor->done=GLOBUS_TRUE;
break;
caseGLOBUS_GRAM_PROTOCOL_JOB_STATE_DONE:
if(monitor->verbose)
{
globus_libc_printf("GLOBUS_GRAM_PROTOCOL_JOB_STATE_DONE\n");
}
monitor->done
=GLOBUS_TRUE; break; } globus_cond_signal(&monitor->cond); globus_mutex_unlock(&monitor->mutex);
} 48 Compare:CJobSubmissionExample(cont.) globus_l_globusrun_gramrun(char*request_string,unsignedchar*callback_contact=GLOBUS_NULL;char*job_contact=GLOBUS_NULL;globus_i_globusrun_gram_monitor_tmonitor;interr;monitor.done=GLOBUS_FALSE;monitor.verbose=verbose;globus_mutex_init(&monitor.mutex,GLOBUS_NULL);globus_cond_init(&monitor.cond,GLOBUS_NULL); long options, char *rm_contact){ err=globus_module_activate(GLOBUS_GRAM_CLIENT_MODULE);if(err!
=GLOBUS_SUCCESS){…}err=globus_gram_client_callback_allow( globus_l_globusrun_gram_callback_func,(void*)&monitor,&callback_contact);if(err!
=GLOBUS_SUCCESS){…}err=globus_gram_client_job_request(rm_contact,request_string,GLOBUS_GRAM_PROTOCOL_JOB_STATE_ALL, callback_contact,&job_contact);if(err!
=GLOBUS_SUCCESS) {…}globus_mutex_lock(&monitor.mutex); while(!
monitor.done){globus_cond_wait(&monitor.cond,&monitor.mutex); }globus_mutex_unlock(&monitor.mutex); globus_gram_client_callback_disallow(callback_contact);
globus_free(callback_contact); gglloobbuuss__mcuotnedx__ddeessttrrooyy((&&mmoonniittoorr..cmountde)x;);
49 PersonalGatekeeper Eg:.globus.gatekeeper.Gatekeeper•Runningthegatekeeper –Ant:ant-fdemos.xmlserver–Commandline:globus-personal-gatekeeper(Remembertosettheenvironmentvariables)UsetheGRAMcontactreturned. •Submittingjobstothisgatekeeper –globusrun-o-r"localhost:2119:/O=Grid/O=Globus/OU=mcs.anl.gov/CN=GregorvonLaszewski""&(executable=/c:/cygwin/bin/ls.exe)" 50 RemoteFileTransfer •Basedontheinterfacewecanmovethefilesinthefollowingways. –GUIBased:FileTransferComponentforGrids–Non-GUIBased:CommandLinetools–UsingAPI:ForProgrammers •Basedontheprotocolsitsupports,wecanclassifydifferentwaysasfollow –VanillaFTP:FTP–GridFileTransferProtocol:GridFTP–UrlCopy:HTTP,HTTPS,FTP,andFILE–GlobalesstoSecondaryStorage-GASS: HTTP,HTTPS 51 JavaUrlCopy:copyfrom->to Eg:.globus.io.urlcopy.UrlCopyTest.globus.io.urlcopy.* UrlCopyc=newUrlCopy(); Settingtheurls c.setSourceUrl(from);c.setDestinationUrl(to);c.setUseThirdPartyCopy(true); Registeringwiththelistener c.addUrlCopyListener(newUrlCopyListener(){publicvoidtransfer(inttotal,intcurrent){System.out.println(total+""+current);}publicvoidtransferError(Exceptione){System.out.println("transferfailed:"+e.getMessage());}publicvoidtransferCompleted(){System.out.println("pletedessfully");} }); Transferingthefile c.run(); 52 JavaGridFTPClient BasicFunctions: .globus.ftp.test.GridFTPClient2PartyTest.globus.ftp.*;GridFTPClientclient=newGridFTPClient(host,port);StringfullLocalFile=localDir+"/"+localFile;StringfullRemoteFile=remoteDestDir+"/"+localFile;client.authenticate(GlobusProxy.getDefaultUserProxy());client.setProtectionBufferSize(16384);client.setType(GridFTPSession.TYPE_IMAGE);client.setMode(GridFTPSession.MODE_EBLOCK);client.setDataChannelAuthentication(dcau);client.setDataChannelProtection(prot);DataSourcesource=newDataSourceStream(newFileInputStream(fullLocalFile));client.put(fullRemoteFile,source,null);DataSinksink=newDataSinkStream(newFileOutputStream(fullLocalFile));client.get(fullRemoteFile,sink,null); 53 PythonGridFTPExample frompyGlobusimportftpClientfrompyGlobus.utilimportBufferhandleAttr=ftpClient.HandleAttr()opAttr=ftpClient.OperationAttr()marker=ftpClient.RestartMarker()ftpClnt=ftpClient.FtpClient(handleAttr)ftpClnt.get(url,opAttr,marker,done_func,condV)buf=Buffer(64*1024)handle=ftpClnt.register_read(buf,data_func,0) defdata_func(cv,handle,buffer,bufHandle,bufLen,offset,eof,error):g_dest.write(buffer)ifnoteof:try:handle=g_ftpClient.register_read(g_buffer,data_func,0)exceptException,e: 54 PerformanceOptionsforGridFTP •SettingtcpbuffersizefrompyGlobusimportftpControlbattr=ftpControl.TcpBuffer()battr.set_fixed(64*1024) Or battr.set_automatic(16*1024,8*1024,64*1024)opAttr.set_tcp_buffer(battr)•Settingparallelismpara=ftpControl.Parallelism()para.set_mode(ftpControl.PARALLELISM_FIXED)para.set_size(3)opAttr.set_parallelism(para) 55 PythonGassCopy •ProvidesaprotocolindependentAPItotransferremotefiles. srcAttr =GassCopyAttr() handleAttr=GassCopyHandleAttr() destAttr =GassCopyAttr() ftpSrcAttr=FtpOperationAttr() ftpDestAttr=FtpOperationAttr() srcAttr.set_ftp(ftpSrcAttr) destAttr.set_ftp(ftpDestAttr) copy=GassCopy(handleAttr) copy.copy_url_to_url(srcUrl,srcAttr,destUrl, destAttr) 56 JavaWebBasedPortal Features •Noinstallationneededonuser’sside.Onlyabrowserisrequired.•Ontheportalserver,simpleinstallationprovidedforGridjobportletsusingApacheAnt (/ant).•Portletsallowtheusertosubmitinteractive/batchjobstoGlobusgatekeepers,monitortheirstatus (active,pendingetc.)andviewtheoutput/errors.esstotheGridisbasedonGSIsecurity. SoftwareneededtorunaJetspeedportalserver: •AJavaServletEngine/Container.MustpatiblewiththeServlet2.2orServlet2.3API.Forexample,Tomcat(/tomcat) •JDK1.3orhigherJavaVirtualMachine. RequirementsforGridjobportlets: •JavaCoG•AMyproxyserveronwhichdelegatedusercredentialsarestored.ThisisneededforGSI authentication.•MyProxycredentialessportletdevelopedbyIndianaUniversityExtreme!
ComputingLab. 57 JetspeedPortletforJobSubmission 58 •JavaCoGKit FuturePlans –isbeingheavilyusedinGT3 –formsthebasisofthenewIBMGridApplicationFrameworkforJava(GAF4J) •/tech/gaf4j?
Open&ca=dawhp-pr –WearedevelopinganapacheJetspeedbasedPortaltoolkitforGrids. –WillbedistributedaspartofGT3andseparately •PythonCogKit –WillprovidetheunderlyingsecurityimplementationforthePythonOGSIenvironment –continuetosupportourhigh-levelinterfacesforGT3 59 Acknowledgement •TheJavaCoGEffortispartoftheGlobusProject•TheJavaandPythonCoGKitsarefundedbythe
U.S.DepartmentofEnergyOfficeofScience•SomeJavaCoGworkissupportedbyNSF/Alliance•Moreinformationcanbefoundat ––/cog–http://www-itg.lbl.gov/gtg/projects/pyGlobus/ •Email: –krjackson@lbl.gov–gregor@mcs.anl.gov 60
1 Objectives •WhataretheCoGKits,andwhydoIwanttousethem?
•HowdoIusetheCoGKits?
•HowdoIdevelopmyponentswith theCoGKits?
•Future
2 MotivationforCoGKits •Problem –ManyapplicationdevelopersdesiretoprogramtheGridinframeworksfamiliartothemtoforexampleenablerapidprototyping. •Solution –TheCoGKitprojectintegratesGridsoftwarebasedontheGlobusToolkitandmodityframeworksuchasJavaorPython. •EasierdevelopmentofadvancedGridservices•Easierandmorerapidapplicationdevelopment•EasierdeploymentofGridservices•Codereuseanduseponentrepositories•UseofWebservicesaspartoftheGrids•WidespreaduseoftheGrid
3 CoGsaremore… •CoGsaremorethanjustaninterfacetotheGlobusToolkit •CoGsallowGridprogrammerstousetheCommodityTechnologiesANDtheGridsadvantages –Example:EventandexceptionmodelofJava–Example:SWIGwrappersinPythonforrapid prototypingandlegacycodesupport. •Thus,CoGsarenotjustanAPIbutprovideesstotheCommodityFramework
4 ReasonsforusingtheJavaCoGKit •WhyusetheJavaCoGKit?
–keepingupwithpatchesandprotocolchangesinGlobushasbeendifficultinthepast –TheJavaCoGKithassofarprovidedmunitywitha“service”tohidetheburdenofchangingyourcode. •AskforatestimonybytheXCATgroupatIndianaUniversity. –ManyusersneedonlythefeaturesprovidedintheJavaCoGKit–Little/nochangesweresofarinvolvedtoswitchbetweendifferent versionsoftheGlobusToolkit–Mostlyclientsideprogramming–JavaPlatform: •HighlevelframeworkbettersuitedforGridprogrammingthanC •WhynotusetheJavaCoGKit –ItdoesnotprovideallthelatestfeaturesoftheGlobusToolkit.–ItusesJava,andsomeinmunity…;-)
5 ReasonsforUsingthePythonCoGKit •WhyusethePythonCoGKit?
–ProvidesafullinterfacetotheGlobusToolkit–Little/nochangesareinvolvedinswitchingbetween differentversionsoftheGlobusToolkit–HighlevellanguageallowsforeasierGridprogramming •SupportsrapidprototypingofGridservices/applications –ManyautomatedtoolsexistforexposinglegacyC/C++orFortrancodesasPythonobjects •WhynotusethePythonCoGKit –Smallperformancepenaltyforusinganyinterpretedlanguage •MinimizedbecausethePythonCoGKitisathinwrapperoverthenativeCcode. –Nostatictypechecking–ItusesPython,andsomeinmunity…;-)
6 Motivation:JavaandPythonCoGKits •UseandleverageexistingtechnologiesforGridprogramming-ThecapabilitiesoftheframeworkontowhichGridServicesaremappedcanbeexploited:Objects,Events,Exceptions,JNDI,...-Objectslikejobs/taskscanbedefined.-XMLsupportisprovided.-GUI's,....,IDE'scanbeused(Forte,BOAConstructor…) •Maximizesoftwareflexibility,extensibility,andreusability•Providefoundationsforapplicationdeveloperteamsthatare familiartodevelopapplicationsinthisframework-Reducedevelopmentandmaintenancecost •Useasglueformanytechnologies•Pythoniswellsuitedtotyingtogethermanydifferentlanguages/technologies
7 WhatistheJavaCoGKit?
•TheJavaCoGKitprovidesamappingbetweenJavaandtheGlobusToolkit.ItextendstheuseofGlobusbyenablingtoessadvancedJavafeaturessuchaseventsandobjectsforGridprogramming. •TheJavaCoGKitisimplementedinpureJava.ItspeakstheGridprotocols. •ItisnotawrapperoftheCGlobusToolkit•Thisallowsintegrationwithinapplets.•Mostlyclientsidesupport
8 WhatisthePythonCoGKit?
•SimilarlythePythonCoGKitprovidesamappingbetweenPythonandtheGlobusToolkit.ItextendstheuseofGlobusbyenablingtoessadvancedPythonfeaturessuchasexceptionsandobjectsforGridprogramming. •ThePythonCoGKitisimplementedasaseriesofPythonextensionmodulesthatwraptheGlobusCcode. •UsesSWIG()tohelpgeneratetheinterfaces.
9 Status:JavaCoGKit •Modifiedcoreponents(Protocols) •Basicservicesareprovidedessing: –Security (GSI) –Remotejobsubmissionandmonitoring(GRAM) –RemoteDataess (GridFTP) –InformationServiceess (MDS) –Certificatestore (myProxy) •Current100%clientponentsincludes •ReusableGridponents •AvarietyofGridInterfaces 10 Status:PythonCoGKit •Basicservicesareprovidedessing: –Security (security) –Remotejobsubmissionandmonitoring(gramClient) –SecureworkIO (io) –Protocolindependentdatatransfers (gassCopy) –HighperformanceGridFTPtransfers (ftpClient) –SupportforbuildingGridFTPservers (ftpControl) –RemotefileIO (gassFile) –InformationServicesess •Highlevelservicesforeasierusage •Taskbasedservicestomonusagepatterns 11 Communication •WebPage –/cog–www-itg.lbl.gov/gtg/projects/pyGlobus/ •Bugs –Java: •/cog/contact/bugs/IsmaintainedbytheGlobusProjectbutdoesnotcontainsomeofthements •http://arbat.mcs.anl.gov:8080/bugzilla/Isouroldbugzilla –Python: •http://www-itg.lbl.gov/bugzilla/ •Download –/cog–/cog/java/–www-itg.lbl.gov/gtg/projects/download/download_info.html 12 Versions •JavaVersions –0.9.13isapatiblewithGlobus2.0 •wemendusingGlobus2.0ifyouliketousethisrelease –1.1apatiblewithGlobusToolkit2.2andinfutureGlobusToolkit3.0,e.g.muchofitmaybedistributedaspartofGlobusToolkit3.0 •WemendusingthisversionforGlobus2.2•Stillanalpharelease •PythonVersions –0.9.8 •Twoversions,oneforGT2.2andoneforGT2.0 13 Download&Compile 14 JavaCoGSetupOptions •DownloadSourceorbinary –/cog •Binary –ReadtheReadme–setup–changetheenvironmentvariables •Source –Readthereadme–setup–changetheenvironmentvariables •Manualwillbeavailableshortly 15 PythonCoGKitSetupOptions •Downloadsource –SetEnvironmentvariables–Compile–Install •EasytocreateRPM’sforLinuxsystems•Binaryingforwin32 –AssoonasGlobusofficiallyreleasesthewin32port •Canbuildbinarypackagesforanyplatform•UsesthestandardPythondistutilmodule 16 Requirements •JavaCoG –JDK1.3.1or1.4.1 • –ant •/ant/ •PythonCoG –Python2.0+ • –GlobusToolkitInstallation–GPTInstallation 17 JavaCoGCompilation •cdwork •wecreateabuild •cdogce•antdist•cd..•
=GLOBUS_TRUE; break; } globus_cond_signal(&monitor->cond); globus_mutex_unlock(&monitor->mutex);
} 48 Compare:CJobSubmissionExample(cont.) globus_l_globusrun_gramrun(char*request_string,unsignedchar*callback_contact=GLOBUS_NULL;char*job_contact=GLOBUS_NULL;globus_i_globusrun_gram_monitor_tmonitor;interr;monitor.done=GLOBUS_FALSE;monitor.verbose=verbose;globus_mutex_init(&monitor.mutex,GLOBUS_NULL);globus_cond_init(&monitor.cond,GLOBUS_NULL); long options, char *rm_contact){ err=globus_module_activate(GLOBUS_GRAM_CLIENT_MODULE);if(err!
=GLOBUS_SUCCESS){…}err=globus_gram_client_callback_allow( globus_l_globusrun_gram_callback_func,(void*)&monitor,&callback_contact);if(err!
=GLOBUS_SUCCESS){…}err=globus_gram_client_job_request(rm_contact,request_string,GLOBUS_GRAM_PROTOCOL_JOB_STATE_ALL, callback_contact,&job_contact);if(err!
=GLOBUS_SUCCESS) {…}globus_mutex_lock(&monitor.mutex); while(!
monitor.done){globus_cond_wait(&monitor.cond,&monitor.mutex); }globus_mutex_unlock(&monitor.mutex); globus_gram_client_callback_disallow(callback_contact);
globus_free(callback_contact); gglloobbuuss__mcuotnedx__ddeessttrrooyy((&&mmoonniittoorr..cmountde)x;);
49 PersonalGatekeeper Eg:.globus.gatekeeper.Gatekeeper•Runningthegatekeeper –Ant:ant-fdemos.xmlserver–Commandline:globus-personal-gatekeeper(Remembertosettheenvironmentvariables)UsetheGRAMcontactreturned. •Submittingjobstothisgatekeeper –globusrun-o-r"localhost:2119:/O=Grid/O=Globus/OU=mcs.anl.gov/CN=GregorvonLaszewski""&(executable=/c:/cygwin/bin/ls.exe)" 50 RemoteFileTransfer •Basedontheinterfacewecanmovethefilesinthefollowingways. –GUIBased:FileTransferComponentforGrids–Non-GUIBased:CommandLinetools–UsingAPI:ForProgrammers •Basedontheprotocolsitsupports,wecanclassifydifferentwaysasfollow –VanillaFTP:FTP–GridFileTransferProtocol:GridFTP–UrlCopy:HTTP,HTTPS,FTP,andFILE–GlobalesstoSecondaryStorage-GASS: HTTP,HTTPS 51 JavaUrlCopy:copyfrom->to Eg:.globus.io.urlcopy.UrlCopyTest.globus.io.urlcopy.* UrlCopyc=newUrlCopy(); Settingtheurls c.setSourceUrl(from);c.setDestinationUrl(to);c.setUseThirdPartyCopy(true); Registeringwiththelistener c.addUrlCopyListener(newUrlCopyListener(){publicvoidtransfer(inttotal,intcurrent){System.out.println(total+""+current);}publicvoidtransferError(Exceptione){System.out.println("transferfailed:"+e.getMessage());}publicvoidtransferCompleted(){System.out.println("pletedessfully");} }); Transferingthefile c.run(); 52 JavaGridFTPClient BasicFunctions: .globus.ftp.test.GridFTPClient2PartyTest.globus.ftp.*;GridFTPClientclient=newGridFTPClient(host,port);StringfullLocalFile=localDir+"/"+localFile;StringfullRemoteFile=remoteDestDir+"/"+localFile;client.authenticate(GlobusProxy.getDefaultUserProxy());client.setProtectionBufferSize(16384);client.setType(GridFTPSession.TYPE_IMAGE);client.setMode(GridFTPSession.MODE_EBLOCK);client.setDataChannelAuthentication(dcau);client.setDataChannelProtection(prot);DataSourcesource=newDataSourceStream(newFileInputStream(fullLocalFile));client.put(fullRemoteFile,source,null);DataSinksink=newDataSinkStream(newFileOutputStream(fullLocalFile));client.get(fullRemoteFile,sink,null); 53 PythonGridFTPExample frompyGlobusimportftpClientfrompyGlobus.utilimportBufferhandleAttr=ftpClient.HandleAttr()opAttr=ftpClient.OperationAttr()marker=ftpClient.RestartMarker()ftpClnt=ftpClient.FtpClient(handleAttr)ftpClnt.get(url,opAttr,marker,done_func,condV)buf=Buffer(64*1024)handle=ftpClnt.register_read(buf,data_func,0) defdata_func(cv,handle,buffer,bufHandle,bufLen,offset,eof,error):g_dest.write(buffer)ifnoteof:try:handle=g_ftpClient.register_read(g_buffer,data_func,0)exceptException,e: 54 PerformanceOptionsforGridFTP •SettingtcpbuffersizefrompyGlobusimportftpControlbattr=ftpControl.TcpBuffer()battr.set_fixed(64*1024) Or battr.set_automatic(16*1024,8*1024,64*1024)opAttr.set_tcp_buffer(battr)•Settingparallelismpara=ftpControl.Parallelism()para.set_mode(ftpControl.PARALLELISM_FIXED)para.set_size(3)opAttr.set_parallelism(para) 55 PythonGassCopy •ProvidesaprotocolindependentAPItotransferremotefiles. srcAttr =GassCopyAttr() handleAttr=GassCopyHandleAttr() destAttr =GassCopyAttr() ftpSrcAttr=FtpOperationAttr() ftpDestAttr=FtpOperationAttr() srcAttr.set_ftp(ftpSrcAttr) destAttr.set_ftp(ftpDestAttr) copy=GassCopy(handleAttr) copy.copy_url_to_url(srcUrl,srcAttr,destUrl, destAttr) 56 JavaWebBasedPortal Features •Noinstallationneededonuser’sside.Onlyabrowserisrequired.•Ontheportalserver,simpleinstallationprovidedforGridjobportletsusingApacheAnt (/ant).•Portletsallowtheusertosubmitinteractive/batchjobstoGlobusgatekeepers,monitortheirstatus (active,pendingetc.)andviewtheoutput/errors.esstotheGridisbasedonGSIsecurity. SoftwareneededtorunaJetspeedportalserver: •AJavaServletEngine/Container.MustpatiblewiththeServlet2.2orServlet2.3API.Forexample,Tomcat(/tomcat) •JDK1.3orhigherJavaVirtualMachine. RequirementsforGridjobportlets: •JavaCoG•AMyproxyserveronwhichdelegatedusercredentialsarestored.ThisisneededforGSI authentication.•MyProxycredentialessportletdevelopedbyIndianaUniversityExtreme!
ComputingLab. 57 JetspeedPortletforJobSubmission 58 •JavaCoGKit FuturePlans –isbeingheavilyusedinGT3 –formsthebasisofthenewIBMGridApplicationFrameworkforJava(GAF4J) •/tech/gaf4j?
Open&ca=dawhp-pr –WearedevelopinganapacheJetspeedbasedPortaltoolkitforGrids. –WillbedistributedaspartofGT3andseparately •PythonCogKit –WillprovidetheunderlyingsecurityimplementationforthePythonOGSIenvironment –continuetosupportourhigh-levelinterfacesforGT3 59 Acknowledgement •TheJavaCoGEffortispartoftheGlobusProject•TheJavaandPythonCoGKitsarefundedbythe
U.S.DepartmentofEnergyOfficeofScience•SomeJavaCoGworkissupportedbyNSF/Alliance•Moreinformationcanbefoundat ––/cog–http://www-itg.lbl.gov/gtg/projects/pyGlobus/ •Email: –krjackson@lbl.gov–gregor@mcs.anl.gov 60
声明:
该资讯来自于互联网网友发布,如有侵犯您的权益请联系我们。