Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4024 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.7951. Min Range: -2.7951

Max Act: 2.7951. Min Act: -0.0000

Data Index: 70059 (C4 (Web Text))

Max Activating Token Index: 293

Text #1

Max Range: 2.7951. Min Range: -2.7951

Max Act: 2.7663. Min Act: -0.0000

Data Index: 589424 (C4 (Web Text))

Max Activating Token Index: 465

Text #2

Max Range: 2.7951. Min Range: -2.7951

Max Act: 2.7248. Min Act: -0.0000

Data Index: 1134323 (C4 (Web Text))

Max Activating Token Index: 951

Text #3

Max Range: 2.7951. Min Range: -2.7951

Max Act: 2.6814. Min Act: -0.0000

Data Index: 253834 (C4 (Web Text))

Max Activating Token Index: 591

Text #4

Max Range: 2.7951. Min Range: -2.7951

Max Act: 2.6366. Min Act: -0.0000

Data Index: 908263 (C4 (Web Text))

Max Activating Token Index: 77

Text #5

Max Range: 2.7951. Min Range: -2.7951

Max Act: 2.7628. Min Act: -0.0000

Data Index: 74273 (C4 (Web Text))

Max Activating Token Index: 1007

Text #6

Max Range: 2.7951. Min Range: -2.7951

Max Act: 2.4792. Min Act: -0.0000

Data Index: 20603 (C4 (Web Text))

Max Activating Token Index: 863

Text #7

Max Range: 2.7951. Min Range: -2.7951

Max Act: 2.7256. Min Act: -0.0000

Data Index: 93840 (C4 (Web Text))

Max Activating Token Index: 855

Text #8

Max Range: 2.7951. Min Range: -2.7951

Max Act: 2.5383. Min Act: -0.0000

Data Index: 151899 (C4 (Web Text))

Max Activating Token Index: 942

Text #9

Max Range: 2.7951. Min Range: -2.7951

Max Act: 2.3965. Min Act: -0.0000

Data Index: 853989 (C4 (Web Text))

Max Activating Token Index: 317

Text #10

Max Range: 2.7951. Min Range: -2.7951

Max Act: 2.4587. Min Act: -0.0000

Data Index: 1344683 (C4 (Web Text))

Max Activating Token Index: 529

Text #11

Max Range: 2.7951. Min Range: -2.7951

Max Act: 2.4562. Min Act: -0.0000

Data Index: 1055749 (C4 (Web Text))

Max Activating Token Index: 149

Text #12

Max Range: 2.7951. Min Range: -2.7951

Max Act: 2.4392. Min Act: -0.0000

Data Index: 560429 (C4 (Web Text))

Max Activating Token Index: 516

Text #13

Max Range: 2.7951. Min Range: -2.7951

Max Act: 2.5258. Min Act: -0.0000

Data Index: 1152489 (C4 (Web Text))

Max Activating Token Index: 209

Text #14

Max Range: 2.7951. Min Range: -2.7951

Max Act: 2.3982. Min Act: -0.0000

Data Index: 1224791 (C4 (Web Text))

Max Activating Token Index: 990

Text #15

Max Range: 2.7951. Min Range: -2.7951

Max Act: 2.3670. Min Act: -0.0000

Data Index: 240560 (C4 (Web Text))

Max Activating Token Index: 956

Text #16

Max Range: 2.7951. Min Range: -2.7951

Max Act: 2.3769. Min Act: -0.0000

Data Index: 569256 (C4 (Web Text))

Max Activating Token Index: 893

Text #17

Max Range: 2.7951. Min Range: -2.7951

Max Act: 2.2264. Min Act: -0.0000

Data Index: 1279352 (C4 (Web Text))

Max Activating Token Index: 437

Text #18

Max Range: 2.7951. Min Range: -2.7951

Max Act: 2.3629. Min Act: -0.0000

Data Index: 998102 (C4 (Web Text))

Max Activating Token Index: 552

Text #19

Max Range: 2.7951. Min Range: -2.7951

Max Act: 2.3183. Min Act: -0.0000

Data Index: 767603 (C4 (Web Text))

Max Activating Token Index: 596