Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3092 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.6194. Min Range: -0.6194

Max Act: 0.6194. Min Act: -0.0000

Data Index: 899329 (C4 (Web Text))

Max Activating Token Index: 323

Text #1

Max Range: 0.6194. Min Range: -0.6194

Max Act: 0.5550. Min Act: -0.0000

Data Index: 88623 (C4 (Web Text))

Max Activating Token Index: 919

Text #2

Max Range: 0.6194. Min Range: -0.6194

Max Act: 0.4864. Min Act: -0.0000

Data Index: 768790 (C4 (Web Text))

Max Activating Token Index: 516

Text #3

Max Range: 0.6194. Min Range: -0.6194

Max Act: 0.4699. Min Act: -0.0000

Data Index: 220194 (C4 (Web Text))

Max Activating Token Index: 222

Text #4

Max Range: 0.6194. Min Range: -0.6194

Max Act: 0.4939. Min Act: -0.0000

Data Index: 188751 (C4 (Web Text))

Max Activating Token Index: 391

Text #5

Max Range: 0.6194. Min Range: -0.6194

Max Act: 0.4735. Min Act: -0.0000

Data Index: 1048561 (C4 (Web Text))

Max Activating Token Index: 521

Text #6

Max Range: 0.6194. Min Range: -0.6194

Max Act: 0.4661. Min Act: -0.0000

Data Index: 1122545 (C4 (Web Text))

Max Activating Token Index: 81

Text #7

Max Range: 0.6194. Min Range: -0.6194

Max Act: 0.4609. Min Act: -0.0000

Data Index: 1346144 (C4 (Web Text))

Max Activating Token Index: 187

Text #8

Max Range: 0.6194. Min Range: -0.6194

Max Act: 0.4763. Min Act: -0.0000

Data Index: 219701 (C4 (Web Text))

Max Activating Token Index: 596

Text #9

Max Range: 0.6194. Min Range: -0.6194

Max Act: 0.4893. Min Act: -0.0000

Data Index: 47794 (C4 (Web Text))

Max Activating Token Index: 107

Text #10

Max Range: 0.6194. Min Range: -0.6194

Max Act: 0.4482. Min Act: -0.0000

Data Index: 814488 (C4 (Web Text))

Max Activating Token Index: 445

Text #11

Max Range: 0.6194. Min Range: -0.6194

Max Act: 0.4224. Min Act: -0.0000

Data Index: 1103023 (C4 (Web Text))

Max Activating Token Index: 166

Text #12

Max Range: 0.6194. Min Range: -0.6194

Max Act: 0.4382. Min Act: -0.0000

Data Index: 351330 (C4 (Web Text))

Max Activating Token Index: 324

Text #13

Max Range: 0.6194. Min Range: -0.6194

Max Act: 0.4109. Min Act: -0.0000

Data Index: 248335 (C4 (Web Text))

Max Activating Token Index: 24

Text #14

Max Range: 0.6194. Min Range: -0.6194

Max Act: 0.4661. Min Act: -0.0000

Data Index: 257401 (C4 (Web Text))

Max Activating Token Index: 747

Text #15

Max Range: 0.6194. Min Range: -0.6194

Max Act: 0.4173. Min Act: -0.0000

Data Index: 1239402 (C4 (Web Text))

Max Activating Token Index: 72

Text #16

Max Range: 0.6194. Min Range: -0.6194

Max Act: 0.3988. Min Act: -0.0000

Data Index: 869087 (C4 (Web Text))

Max Activating Token Index: 837

Text #17

Max Range: 0.6194. Min Range: -0.6194

Max Act: 0.4088. Min Act: -0.0000

Data Index: 691373 (C4 (Web Text))

Max Activating Token Index: 98

Text #18

Max Range: 0.6194. Min Range: -0.6194

Max Act: 0.3968. Min Act: -0.0000

Data Index: 758810 (C4 (Web Text))

Max Activating Token Index: 69

Text #19

Max Range: 0.6194. Min Range: -0.6194

Max Act: 0.3904. Min Act: -0.0000

Data Index: 183965 (C4 (Web Text))

Max Activating Token Index: 806