= Robotsko pregledovanje in manipulacija tekstila in tkanin (RTFM) = [[https://www.ijs.si/ijsw/ARRSProjekti/2022|Nazaj na seznam za leto 2022]] ---- === Oznaka in naziv projekta === J2-4457 Robotsko pregledovanje in manipulacija tekstila in tkanin (RTFM)<
> J2-4457 Robot Textile and Fabric Inspection and Manipulation (RTFM)<
> === Logotipi ARRS in drugih sofinancerjev === {{https://www.ijs.si/ijsw/ARRSProjekti/SeznamARRSProjekti?action=AttachFile&do=get&target=ARRS_logotip.jpg|© Javna agencija za raziskovalno dejavnost Republike Slovenije|height="150",width="349"}} === Projektna skupina === Vodja projekta: [[https://cris.cobiss.net/ecris/si/sl/researcher/19021|izr. prof. dr. Andrej Gams (25638)]] '''Sodelujoče raziskovalne organizacije: '''{{{#!wiki comment [[https://www.sicris.si/public/jqm/search_basic.aspx?lang=slv&opt=2&subopt=1&opdescr=search&code1=cmn&code2=auto&search_term=N1-0230|Povezava na SICRIS]] }}} '''''[[http://www.ijs.si/|Institut "Jožef Stefan"]], [[http://abr.ijs.si/|Odsek za avtomatiko, biokibernetiko in robotiko]] [[https://cris.cobiss.net/ecris/si/sl/group/1609|(0106-014)]], (IJS)''''' '''''[[https://www.uni-lj.si|Univerza v Ljubljani]], [[https://www.fri.uni-lj.si/|Fakulteta za računalništvo in informatiko]], [[https://www.vicos.si//|Laboratorij za umetne vizualne spoznavne sisteme]], [[https://cris.cobiss.net/ecris/si/sl/group/3077|(1539-011)]], (FRI)''''' '''Sestava projektne skupine: '''{{{#!wiki comment [[https://www.sicris.si/public/jqm/search_basic.aspx?lang=slv&opt=2&subopt=1&opdescr=search&code1=cmn&code2=auto&search_term=N1-0230|Povezava na SICRIS]] }}} * [[https://cris.cobiss.net/ecris/si/sl/researcher/7480|prof. dr. Aleš Ude (11772)]] - IJS * [[https://cris.cobiss.net/ecris/si/sl/researcher/3964|prof. dr. Bojan Nemec (118)]] - IJS * [[https://cris.cobiss.net/ecris/si/sl/researcher/50228|Zvezdan Lončarević (53767)]] - IJS * [[https://cris.cobiss.net/ecris/si/sl/researcher/52613|Peter Nimac (55794)]] - IJS * [[https://cris.cobiss.net/ecris/si/sl/researcher/47478|Matija Mavsar (51232)]] - IJS * [[https://cris.cobiss.net/ecris/si/sl/researcher/45281|Simon Reberšek (39258)]] - IJS * [[https://cris.cobiss.net/ecris/si/sl/researcher/10425|dr. Danijel Skočaj (18198)]] - FRI * [[https://cris.cobiss.net/ecris/si/sl/researcher/39006|dr. Domen Tabernik (34398)]] - FRI * [[https://cris.cobiss.net/ecris/si/sl/researcher/50400|Vitjan Zavrtanik (53924)]] - FRI === Vsebinski opis projekta === Vsebinski opis projekta '''Manipulacija tekstila in tkanin''' oz. blaga je pomembno področje robotskih raziskav z aplikacijami tako v industriji kot v domovih. Vendar pa je napredek pri '''robotski manipulaciji takih deformabilnih predmetov zaostal za napredkom manipulacije togih predmetov''' zaradi veliko bolj '''zapletene dinamike in konfiguracijskega prostora'''. V tem projektu bomo uporabili nove, '''napredne metode globokega učenja in prenosa simulacije v realnost''' za reševanje praktičnega in obstoječega problema '''robotskega pregledovanja in manipulacije tekstila in tkanin'''. Nadgradili bomo najsodobnejše metode '''zaznavanja/pregledovanja''' ter '''robotske manipulacije''' tekstila in blaga ter '''premostili tehnološko vrzel''', ki onemogoča avtomatizacijo tovrstnega ravnanja z materiali. Tehnološki napredek bomo '''demonstrirali '''na stopnji tehnološke zrelosti '''TRL 4''' – tehnologija, prikazana v laboratoriju. Rezultati tega projekta bodo '''osnova za potencialne prihodnje''', aplikativne izvedbe, ki bodo '''povečale konkurenčnost''' slovenskih in evropskih podjetij, ki se ukvarjajo s proizvodnjo in logistiko izdelkov iz blaga in tekstila. V okviru projekta bomo '''reševali '''naštete probleme 1) Razvili bomo '''vizualni sistem''', ki omogoča segmentacijo, karakterizacijo in pregled manipuliranega tekstila/tkanine. Temeljil bo na '''robustni večmodalni segmentaciji na osnovi globokega učenja''' in zaznavanju ključnih točk, pomembnih za prijemanje ter na nenadzorovanem učenju za odkrivanje napak. 2) Razvili in prikazali bomo '''učinkovito ciljno usmerjeno manipulacijo''' tekstilov/tkanin. To bomo doseženo z učenjem ustreznih načinov gibanja, kjer bomo za učenje uporabili napredno '''globoko ter spodbujevano učenje''' v simulaciji in v resničnem svetu, '''metode simulacija-resničnost''' in trenirali nove, celostne '''nevronske mreže od vida do gibanja''' 3) Razvili bomo metode za '''načrtovanje (planiranje) zaporedja dejanj''', ki temeljijo na '''novi predstavitvi stanja kosa tekstila oz. blaga'''. Uporabljal se bo za generiranje grafov stanj, kjer bo vsaka povezava predstavljala prehod med stanjema vključno s potrebno robotsko akcijo za izvedbo takega prehoda. Za prikaz tehnološkega napredka bomo implementirali '''dvoročno robotsko celico za logistiko blaga in tekstila na nivoju tehnološke zrelosti TRL4'''. Celica bo '''zaznavala, gladila, pregledovala in zlagala kose tekstila''' – oblačil oz. blaga v želeno, ciljno zloženo stanje. Demonstracija bo zajemala vse glavne vidike manipulacije ter zaznavanja oz. pregledovanja tekstila. '''Textile and fabric manipulation''' is an important area of robotics research that has applications both in the industry and in homes. Yet, advances in robotic '''manipulation of such deformable objects have lagged behind work on rigid objects''' due to the far more '''complex dynamics and configuration space'''. In this project we will apply novel,''' advanced deep-learning and sim-to-real transfer learning '''methods on a''' '''real-world problem of '''textile and fabric manipulation and inspection'''. We will advance the state of the art of '''perception/inspection''' and '''robotic manipulation '''of''' '''textile and fabric, in order to '''bridge the technological gap''' and enable automation of such material''' '''handling. We will '''demonstrate''' technological advances at technology readiness level '''TRL 4''' – technology demonstrated in lab. The outcomes of this project will serve as '''foundation for future''', applied implementations, which will '''increase''' the '''competitiveness''' of Slovenian and European companies that deal with production and logistics of textile and fabric items. Through the course of the project we will '''solve''' these problems. We will 1) develop a '''vision-based system''' that allows segmentation, characterization and inspection of the manipulated textiles/fabrics. It will be based on '''robust deep-learning-based''' multimodal segmentation and detection of key points relevant for grasping, as well as on unsupervised learning for defect detection. 2) We will develop and demonstrate '''effective goal directed handling and manipulation of''' textile/fabric objects. This will be achieved through learning of appropriate motion policies, where advanced '''deep learning and reinforcement learning methods''' in simulation and in the real world, '''sim-to-real''' methods and training of new, end-to-end '''vision-to-motion deep neural networks''' will be applied. 3) Finally, we will develop means to '''plan a sequence of actions '''based on a '''novel state representation of textile'''. It will be used to form graphs of states where each edge is a transition with information on the needed robot action. To demonstrate the technological advances, we will implement a '''bimanual robot cell''' for '''textile '''and '''fabric logistics at TRL4'''. It will '''detect, flatten, inspect and fold textiles and fabrics '''into desired goal states. The presented demonstration will cover all the major aspects of the project in perception/inspection and handling/manipulation of such deformable objects. Osnovni podatki sofinanciranja so dostopni na spletni strani [[http://www.sicris.si/|SICRIS]]. === Faze projekta in opis njihove realizacije === 1. Faza - Razvoj vizualnega sistema, ki omogoča segmentacijo, karakterizacijo in pregled manipuliranega tekstila/tkanine. 2. Faza - Razvoj in prikaz učinkovite ciljno usmerjene manipulacije z uporabo metod globokega učenja. 3. Faza - Razvoj metod načrtovanja zaporedja in prehodov med stanji tekstila. 1. Phase - Development of vision-based system that allows segmentation, characterization and inspection of manipulated textiles/fabrics. 2. Phase - Development and demonstration of effective goal directed handling and manipulation with the use of deep learning methods. 3. Phase - Development of methods to plan and execute a sequence of actions to transition between different states of textile. === Bibliografske reference === [[https://bib.cobiss.net/bibliographies/si/webBiblio/bib351_20240611_112215_j2-4457.html|Reference - SICRIS]] D. Tabernik, J. N. Muhovič, in D. Skočaj, „Dense center-direction regression for object counting and localization with point supervision“, Pattern recogn., let. 153, št. ] 110540, str. 1–13, 2024, [Na spletu]. Dostopno na: [[https://www.sciencedirect.com/science/article/pii/S0031320324002917]]<
> [COBISS.SI-ID - [[https://plus.cobiss.net/cobiss/si/sl/bib/195818755|195818755]] ] D. Tabernik, J. N. Muhovič, in D. Skočaj, „Lokalizacija in ocenjevanje lege predmeta v treh prostostnih stopnjah s središčnimi smernimi vektorji“, 2023, str. 359–362. [Na spletu]. Dostopno na: [[https://erk.fe.uni-lj.si/2023/papers/tabernik(lokalizacija_in).pdf]]<
> [COBISS.SI-ID - [[https://plus.cobiss.net/cobiss/si/sl/bib/167408387|167408387]] ] D. Tabernik, M. Šuc, in D. Skočaj, „Automated detection and segmentation of cracks in concrete surfaces using joined segmentation and classification deep neural network“, Constr. build. mater., let. 408, št. ] 133582, str. 1–12, 2023, [Na spletu]. Dostopno na: [[https://www.sciencedirect.com/science/article/abs/pii/S0950061823033007?via%3Dihub]]<
> [COBISS.SI-ID - [[https://plus.cobiss.net/cobiss/si/sl/bib/168164611|168164611]] ] D. Tabernik, P. Mlakar, J. Božič, L. Čehovin Zajc, V. Rijavec, in D. Skočaj, „Demonstracijska celica za prikaz globokega učenja v praktičnih aplikacijah“, 2024, str. 25–36. [Na spletu]. Dostopno na: [[https://press.um.si/index.php/ump/catalog/view/859/1246/3897]]<
> [COBISS.SI-ID - [[https://plus.cobiss.net/cobiss/si/sl/bib/191481347|191481347]] ] P. Nimac, M. Mavsar, in A. Gams, „Cloth smoothing simulation with vision-to-motion skill model“, 2022, str. 220–223. [Na spletu]. Dostopno na: [[https://erk.fe.uni-lj.si/2022/erk22.pdf]]<
> [COBISS.SI-ID - [[https://plus.cobiss.net/cobiss/si/sl/bib/122679043|122679043]] ] P. Nimac in A. Gams, „Cloth flattening with vision-to-motion skill model“, 2023, str. 367–374. [Na spletu]. Dostopno na: [[https://link.springer.com/chapter/10.1007/978-3-031-32606-6_43]]<
> [COBISS.SI-ID - [[https://plus.cobiss.net/cobiss/si/sl/bib/154215939|154215939]] ] P. Nimac in A. Gams, „Vpliv parametrov barvnega modela pri robotskem ravnanju tekstila“, 2023, str. 189–192. [Na spletu]. Dostopno na: [[https://erk.fe.uni-lj.si/2023/papers/nimac(vpliv_parametrov).pdf]]<
> [COBISS.SI-ID - [[https://plus.cobiss.net/cobiss/si/sl/bib/174868483|174868483]] ] P. Nimac in A. Gams, „Determining Sample Quantity for Robot Vision-to-Motion Cloth Flattening“, 2024, str. 3–11. [Na spletu]. Dostopno na: [[https://link.springer.com/chapter/10.1007/978-3-031-59257-7_1]]<
> [COBISS.SI-ID - [[https://plus.cobiss.net/cobiss/si/sl/bib/197714179|197714179]] ] === Dosežki === ~+'''Zmagovalci''' izziva '''PERCEPTION'''+~ na [[https://www.iri.upc.edu/groups/perception/ClothManipulationChallenge/|2nd Edition Cloth Manipulation and Perception, 7th Robotic Grasping and Manipulation Competition (RGMC) 2023]]<
> '''Domen Tabernik, Matej Urbas, Jon Muhovič in Danijel Skočaj'''<
> ICRA 2023, ViCoS-FRI team from the University of Ljubljana in Slovenia ~+'''2. mesto''' na ICRA 2024 - Cloth Competition+~ [[https://airo.ugent.be/cloth_competition/|Cloth Manipulation track, 9th Robotic Grasping and Manipulation Competition (RGMC) 2024]]<
> '''Domen Tabernik, Andrej Gams, Peter Nimac, Matej Urbas, Jon Muhovič, Danijel Skočaj in Matija Mavsar'''<
> ICRA 2024, Team Ljubljana ~+The '''BEST''' Application '''PAPER FINALIST AWARD'''+~ za prispevek predstavljen na 33. International Conference on Robotics in Alpe-Adria-Danube Region z naslovom ''Determining Sample Quantity for Robot Vision-to-Motion Cloth Flattening'' [[https://raad.utcluj.ro/index.php/general-info/best-paper-awards|RAAD 2024 Best Paper Awards]]<
> '''Peter Nimac in Andrej Gams'''<
> RAAD 2024 ---- [[https://www.ijs.si/ijsw/ARRSProjekti/2022|Nazaj na seznam za leto 2022]]