Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1196 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.5862. Min Range: -0.5862

Max Act: 0.5862. Min Act: -0.0000

Data Index: 402436 (C4 (Web Text))

Max Activating Token Index: 53

Text #1

Max Range: 0.5862. Min Range: -0.5862

Max Act: 0.5688. Min Act: -0.0000

Data Index: 840196 (C4 (Web Text))

Max Activating Token Index: 56

Text #2

Max Range: 0.5862. Min Range: -0.5862

Max Act: 0.5411. Min Act: -0.0000

Data Index: 1322902 (C4 (Web Text))

Max Activating Token Index: 404

Text #3

Max Range: 0.5862. Min Range: -0.5862

Max Act: 0.5587. Min Act: -0.0000

Data Index: 1051029 (C4 (Web Text))

Max Activating Token Index: 66

Text #4

Max Range: 0.5862. Min Range: -0.5862

Max Act: 0.5738. Min Act: -0.0000

Data Index: 282077 (C4 (Web Text))

Max Activating Token Index: 111

Text #5

Max Range: 0.5862. Min Range: -0.5862

Max Act: 0.5207. Min Act: -0.0000

Data Index: 764439 (C4 (Web Text))

Max Activating Token Index: 532

Text #6

Max Range: 0.5862. Min Range: -0.5862

Max Act: 0.5636. Min Act: -0.0000

Data Index: 226717 (C4 (Web Text))

Max Activating Token Index: 63

Text #7

Max Range: 0.5862. Min Range: -0.5862

Max Act: 0.5403. Min Act: -0.0000

Data Index: 212517 (C4 (Web Text))

Max Activating Token Index: 126

Text #8

Max Range: 0.5862. Min Range: -0.5862

Max Act: 0.5377. Min Act: -0.0000

Data Index: 92302 (C4 (Web Text))

Max Activating Token Index: 513

Text #9

Max Range: 0.5862. Min Range: -0.5862

Max Act: 0.5388. Min Act: -0.0000

Data Index: 918403 (C4 (Web Text))

Max Activating Token Index: 183

Text #10

Max Range: 0.5862. Min Range: -0.5862

Max Act: 0.4987. Min Act: -0.0000

Data Index: 934603 (C4 (Web Text))

Max Activating Token Index: 498

Text #11

Max Range: 0.5862. Min Range: -0.5862

Max Act: 0.4858. Min Act: -0.0000

Data Index: 1197858 (C4 (Web Text))

Max Activating Token Index: 829

Text #12

Max Range: 0.5862. Min Range: -0.5862

Max Act: 0.5110. Min Act: -0.0000

Data Index: 1077218 (C4 (Web Text))

Max Activating Token Index: 743

Text #13

Max Range: 0.5862. Min Range: -0.5862

Max Act: 0.4750. Min Act: -0.0000

Data Index: 849367 (C4 (Web Text))

Max Activating Token Index: 92

Text #14

Max Range: 0.5862. Min Range: -0.5862

Max Act: 0.5238. Min Act: -0.0000

Data Index: 172220 (C4 (Web Text))

Max Activating Token Index: 72

Text #15

Max Range: 0.5862. Min Range: -0.5862

Max Act: 0.5345. Min Act: -0.0000

Data Index: 500476 (C4 (Web Text))

Max Activating Token Index: 112

Text #16

Max Range: 0.5862. Min Range: -0.5862

Max Act: 0.4957. Min Act: -0.0000

Data Index: 1217463 (C4 (Web Text))

Max Activating Token Index: 570

Text #17

Max Range: 0.5862. Min Range: -0.5862

Max Act: 0.4990. Min Act: -0.0000

Data Index: 1021091 (C4 (Web Text))

Max Activating Token Index: 79

Text #18

Max Range: 0.5862. Min Range: -0.5862

Max Act: 0.5476. Min Act: -0.0000

Data Index: 341425 (C4 (Web Text))

Max Activating Token Index: 873

Text #19

Max Range: 0.5862. Min Range: -0.5862

Max Act: 0.5105. Min Act: -0.0000

Data Index: 685655 (C4 (Web Text))

Max Activating Token Index: 459