Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3405 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.4191. Min Range: -0.4191

Max Act: 0.4191. Min Act: -0.0002

Data Index: 1609670 (Python Code)

Max Activating Token Index: 794

Text #1

Max Range: 0.4191. Min Range: -0.4191

Max Act: 0.4066. Min Act: -0.0002

Data Index: 1369860 (Python Code)

Max Activating Token Index: 105

Text #2

Max Range: 0.4191. Min Range: -0.4191

Max Act: 0.3613. Min Act: -0.0002

Data Index: 1558968 (Python Code)

Max Activating Token Index: 535

Text #3

Max Range: 0.4191. Min Range: -0.4191

Max Act: 0.3553. Min Act: -0.0002

Data Index: 1515272 (Python Code)

Max Activating Token Index: 553

Text #4

Max Range: 0.4191. Min Range: -0.4191

Max Act: 0.3663. Min Act: -0.0002

Data Index: 1448562 (Python Code)

Max Activating Token Index: 444

Text #5

Max Range: 0.4191. Min Range: -0.4191

Max Act: 0.3542. Min Act: -0.0002

Data Index: 1405491 (Python Code)

Max Activating Token Index: 265

Text #6

Max Range: 0.4191. Min Range: -0.4191

Max Act: 0.3398. Min Act: -0.0002

Data Index: 1470793 (Python Code)

Max Activating Token Index: 758

Text #7

Max Range: 0.4191. Min Range: -0.4191

Max Act: 0.3342. Min Act: -0.0002

Data Index: 1507835 (Python Code)

Max Activating Token Index: 782

Text #8

Max Range: 0.4191. Min Range: -0.4191

Max Act: 0.3334. Min Act: -0.0002

Data Index: 1478537 (Python Code)

Max Activating Token Index: 90

Text #9

Max Range: 0.4191. Min Range: -0.4191

Max Act: 0.3498. Min Act: -0.0002

Data Index: 1414759 (Python Code)

Max Activating Token Index: 451

Text #10

Max Range: 0.4191. Min Range: -0.4191

Max Act: 0.3093. Min Act: -0.0002

Data Index: 1403495 (Python Code)

Max Activating Token Index: 730

Text #11

Max Range: 0.4191. Min Range: -0.4191

Max Act: 0.3206. Min Act: -0.0002

Data Index: 1475005 (Python Code)

Max Activating Token Index: 32

Text #12

Max Range: 0.4191. Min Range: -0.4191

Max Act: 0.3299. Min Act: -0.0002

Data Index: 1436008 (Python Code)

Max Activating Token Index: 610

Text #13

Max Range: 0.4191. Min Range: -0.4191

Max Act: 0.3294. Min Act: -0.0002

Data Index: 1459637 (Python Code)

Max Activating Token Index: 520

Text #14

Max Range: 0.4191. Min Range: -0.4191

Max Act: 0.3149. Min Act: -0.0002

Data Index: 1400629 (Python Code)

Max Activating Token Index: 161

Text #15

Max Range: 0.4191. Min Range: -0.4191

Max Act: 0.3270. Min Act: -0.0002

Data Index: 1581912 (Python Code)

Max Activating Token Index: 612

Text #16

Max Range: 0.4191. Min Range: -0.4191

Max Act: 0.3272. Min Act: -0.0002

Data Index: 1400145 (Python Code)

Max Activating Token Index: 735

Text #17

Max Range: 0.4191. Min Range: -0.4191

Max Act: 0.3153. Min Act: -0.0002

Data Index: 1516877 (Python Code)

Max Activating Token Index: 991

Text #18

Max Range: 0.4191. Min Range: -0.4191

Max Act: 0.3266. Min Act: -0.0002

Data Index: 1563011 (Python Code)

Max Activating Token Index: 322

Text #19

Max Range: 0.4191. Min Range: -0.4191

Max Act: 0.3120. Min Act: -0.0002

Data Index: 1650402 (Python Code)

Max Activating Token Index: 455