Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2227 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.4311. Min Range: -0.4311

Max Act: 0.4311. Min Act: -0.0000

Data Index: 1640912 (Python Code)

Max Activating Token Index: 30

Text #1

Max Range: 0.4311. Min Range: -0.4311

Max Act: 0.4002. Min Act: -0.0000

Data Index: 1389076 (Python Code)

Max Activating Token Index: 614

Text #2

Max Range: 0.4311. Min Range: -0.4311

Max Act: 0.3741. Min Act: -0.0000

Data Index: 1610449 (Python Code)

Max Activating Token Index: 677

Text #3

Max Range: 0.4311. Min Range: -0.4311

Max Act: 0.4130. Min Act: -0.0000

Data Index: 1482365 (Python Code)

Max Activating Token Index: 359

Text #4

Max Range: 0.4311. Min Range: -0.4311

Max Act: 0.3784. Min Act: -0.0000

Data Index: 1620422 (Python Code)

Max Activating Token Index: 804

Text #5

Max Range: 0.4311. Min Range: -0.4311

Max Act: 0.3790. Min Act: -0.0000

Data Index: 1644969 (Python Code)

Max Activating Token Index: 183

Text #6

Max Range: 0.4311. Min Range: -0.4311

Max Act: 0.3545. Min Act: -0.0000

Data Index: 1515730 (Python Code)

Max Activating Token Index: 135

Text #7

Max Range: 0.4311. Min Range: -0.4311

Max Act: 0.3670. Min Act: -0.0000

Data Index: 1538881 (Python Code)

Max Activating Token Index: 605

Text #8

Max Range: 0.4311. Min Range: -0.4311

Max Act: 0.3595. Min Act: -0.0000

Data Index: 1557067 (Python Code)

Max Activating Token Index: 180

Text #9

Max Range: 0.4311. Min Range: -0.4311

Max Act: 0.3416. Min Act: -0.0000

Data Index: 1464006 (Python Code)

Max Activating Token Index: 436

Text #10

Max Range: 0.4311. Min Range: -0.4311

Max Act: 0.3768. Min Act: -0.0000

Data Index: 1636200 (Python Code)

Max Activating Token Index: 804

Text #11

Max Range: 0.4311. Min Range: -0.4311

Max Act: 0.3378. Min Act: -0.0000

Data Index: 1615988 (Python Code)

Max Activating Token Index: 371

Text #12

Max Range: 0.4311. Min Range: -0.4311

Max Act: 0.3606. Min Act: -0.0000

Data Index: 1428559 (Python Code)

Max Activating Token Index: 548

Text #13

Max Range: 0.4311. Min Range: -0.4311

Max Act: 0.3587. Min Act: -0.0000

Data Index: 1532149 (Python Code)

Max Activating Token Index: 263

Text #14

Max Range: 0.4311. Min Range: -0.4311

Max Act: 0.3258. Min Act: -0.0000

Data Index: 1480774 (Python Code)

Max Activating Token Index: 835

Text #15

Max Range: 0.4311. Min Range: -0.4311

Max Act: 0.3293. Min Act: -0.0000

Data Index: 1512294 (Python Code)

Max Activating Token Index: 371

Text #16

Max Range: 0.4311. Min Range: -0.4311

Max Act: 0.3432. Min Act: -0.0000

Data Index: 1590129 (Python Code)

Max Activating Token Index: 91

Text #17

Max Range: 0.4311. Min Range: -0.4311

Max Act: 0.3448. Min Act: -0.0000

Data Index: 1419189 (Python Code)

Max Activating Token Index: 290

Text #18

Max Range: 0.4311. Min Range: -0.4311

Max Act: 0.3598. Min Act: -0.0000

Data Index: 1460667 (Python Code)

Max Activating Token Index: 656

Text #19

Max Range: 0.4311. Min Range: -0.4311

Max Act: 0.3269. Min Act: -0.0000

Data Index: 1637990 (Python Code)

Max Activating Token Index: 957