Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1385 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.4343. Min Range: -1.4343

Max Act: 1.4343. Min Act: -0.0000

Data Index: 945333 (C4 (Web Text))

Max Activating Token Index: 603

Text #1

Max Range: 1.4343. Min Range: -1.4343

Max Act: 1.3976. Min Act: -0.0000

Data Index: 416836 (C4 (Web Text))

Max Activating Token Index: 889

Text #2

Max Range: 1.4343. Min Range: -1.4343

Max Act: 1.3020. Min Act: -0.0000

Data Index: 1015978 (C4 (Web Text))

Max Activating Token Index: 26

Text #3

Max Range: 1.4343. Min Range: -1.4343

Max Act: 1.1724. Min Act: -0.0000

Data Index: 466470 (C4 (Web Text))

Max Activating Token Index: 703

Text #4

Max Range: 1.4343. Min Range: -1.4343

Max Act: 1.4225. Min Act: -0.0000

Data Index: 372173 (C4 (Web Text))

Max Activating Token Index: 628

Text #5

Max Range: 1.4343. Min Range: -1.4343

Max Act: 1.3023. Min Act: -0.0000

Data Index: 1272594 (C4 (Web Text))

Max Activating Token Index: 981

Text #6

Max Range: 1.4343. Min Range: -1.4343

Max Act: 1.0978. Min Act: -0.0000

Data Index: 721919 (C4 (Web Text))

Max Activating Token Index: 324

Text #7

Max Range: 1.4343. Min Range: -1.4343

Max Act: 1.2370. Min Act: -0.0000

Data Index: 790912 (C4 (Web Text))

Max Activating Token Index: 58

Text #8

Max Range: 1.4343. Min Range: -1.4343

Max Act: 1.1788. Min Act: -0.0000

Data Index: 885228 (C4 (Web Text))

Max Activating Token Index: 975

Text #9

Max Range: 1.4343. Min Range: -1.4343

Max Act: 1.2237. Min Act: -0.0000

Data Index: 780073 (C4 (Web Text))

Max Activating Token Index: 957

Text #10

Max Range: 1.4343. Min Range: -1.4343

Max Act: 1.2074. Min Act: -0.0000

Data Index: 835946 (C4 (Web Text))

Max Activating Token Index: 662

Text #11

Max Range: 1.4343. Min Range: -1.4343

Max Act: 1.2068. Min Act: -0.0000

Data Index: 507334 (C4 (Web Text))

Max Activating Token Index: 537

Text #12

Max Range: 1.4343. Min Range: -1.4343

Max Act: 1.1743. Min Act: -0.0000

Data Index: 731420 (C4 (Web Text))

Max Activating Token Index: 214

Text #13

Max Range: 1.4343. Min Range: -1.4343

Max Act: 1.2249. Min Act: -0.0000

Data Index: 1155527 (C4 (Web Text))

Max Activating Token Index: 470

Text #14

Max Range: 1.4343. Min Range: -1.4343

Max Act: 1.1734. Min Act: -0.0000

Data Index: 857276 (C4 (Web Text))

Max Activating Token Index: 878

Text #15

Max Range: 1.4343. Min Range: -1.4343

Max Act: 1.1865. Min Act: -0.0000

Data Index: 716893 (C4 (Web Text))

Max Activating Token Index: 280

Text #16

Max Range: 1.4343. Min Range: -1.4343

Max Act: 1.1523. Min Act: -0.0000

Data Index: 1297332 (C4 (Web Text))

Max Activating Token Index: 42

Text #17

Max Range: 1.4343. Min Range: -1.4343

Max Act: 1.0545. Min Act: -0.0000

Data Index: 811886 (C4 (Web Text))

Max Activating Token Index: 750

Text #18

Max Range: 1.4343. Min Range: -1.4343

Max Act: 1.1443. Min Act: -0.0000

Data Index: 585857 (C4 (Web Text))

Max Activating Token Index: 454

Text #19

Max Range: 1.4343. Min Range: -1.4343

Max Act: 1.0754. Min Act: -0.0000

Data Index: 453487 (C4 (Web Text))

Max Activating Token Index: 217