Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2638 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.2122. Min Range: -0.2122

Max Act: 0.2011. Min Act: -0.0000

Data Index: 419810 (C4 (Web Text))

Max Activating Token Index: 1021

Text #1

Max Range: 0.2122. Min Range: -0.2122

Max Act: 0.2037. Min Act: -0.0000

Data Index: 452371 (C4 (Web Text))

Max Activating Token Index: 584

Text #2

Max Range: 0.2122. Min Range: -0.2122

Max Act: 0.2122. Min Act: 0.0000

Data Index: 328245 (C4 (Web Text))

Max Activating Token Index: 1006

Text #3

Max Range: 0.2122. Min Range: -0.2122

Max Act: 0.1867. Min Act: -0.0000

Data Index: 275460 (C4 (Web Text))

Max Activating Token Index: 123

Text #4

Max Range: 0.2122. Min Range: -0.2122

Max Act: 0.1941. Min Act: -0.0000

Data Index: 471542 (C4 (Web Text))

Max Activating Token Index: 210

Text #5

Max Range: 0.2122. Min Range: -0.2122

Max Act: 0.1778. Min Act: -0.0000

Data Index: 258200 (C4 (Web Text))

Max Activating Token Index: 275

Text #6

Max Range: 0.2122. Min Range: -0.2122

Max Act: 0.1591. Min Act: -0.0000

Data Index: 445616 (C4 (Web Text))

Max Activating Token Index: 361

Text #7

Max Range: 0.2122. Min Range: -0.2122

Max Act: 0.1649. Min Act: -0.0000

Data Index: 990801 (C4 (Web Text))

Max Activating Token Index: 86

Text #8

Max Range: 0.2122. Min Range: -0.2122

Max Act: 0.1609. Min Act: -0.0000

Data Index: 120667 (C4 (Web Text))

Max Activating Token Index: 857

Text #9

Max Range: 0.2122. Min Range: -0.2122

Max Act: 0.1578. Min Act: -0.0000

Data Index: 1033663 (C4 (Web Text))

Max Activating Token Index: 27

Text #10

Max Range: 0.2122. Min Range: -0.2122

Max Act: 0.1593. Min Act: -0.0000

Data Index: 1128962 (C4 (Web Text))

Max Activating Token Index: 948

Text #11

Max Range: 0.2122. Min Range: -0.2122

Max Act: 0.1518. Min Act: -0.0000

Data Index: 1354610 (C4 (Web Text))

Max Activating Token Index: 1007

Text #12

Max Range: 0.2122. Min Range: -0.2122

Max Act: 0.1476. Min Act: -0.0000

Data Index: 2091 (C4 (Web Text))

Max Activating Token Index: 319

Text #13

Max Range: 0.2122. Min Range: -0.2122

Max Act: 0.1535. Min Act: -0.0000

Data Index: 102926 (C4 (Web Text))

Max Activating Token Index: 407

Text #14

Max Range: 0.2122. Min Range: -0.2122

Max Act: 0.1569. Min Act: -0.0000

Data Index: 455890 (C4 (Web Text))

Max Activating Token Index: 108

Text #15

Max Range: 0.2122. Min Range: -0.2122

Max Act: 0.1559. Min Act: -0.0000

Data Index: 912821 (C4 (Web Text))

Max Activating Token Index: 424

Text #16

Max Range: 0.2122. Min Range: -0.2122

Max Act: 0.1494. Min Act: -0.0000

Data Index: 22255 (C4 (Web Text))

Max Activating Token Index: 507

Text #17

Max Range: 0.2122. Min Range: -0.2122

Max Act: 0.1474. Min Act: -0.0000

Data Index: 1077311 (C4 (Web Text))

Max Activating Token Index: 53

Text #18

Max Range: 0.2122. Min Range: -0.2122

Max Act: 0.1527. Min Act: -0.0000

Data Index: 1171089 (C4 (Web Text))

Max Activating Token Index: 79

Text #19

Max Range: 0.2122. Min Range: -0.2122

Max Act: 0.1477. Min Act: -0.0000

Data Index: 70826 (C4 (Web Text))

Max Activating Token Index: 10