Model: SoLU Model: 6 Layers, 3072 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1654 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-6l')

Text #0

Max Range: 0.3215. Min Range: -0.3215

Max Act: 0.3215. Min Act: -0.0001

Data Index: 941484 (C4 (Web Text))

Max Activating Token Index: 98

Text #1

Max Range: 0.3215. Min Range: -0.3215

Max Act: 0.3074. Min Act: -0.0001

Data Index: 970813 (C4 (Web Text))

Max Activating Token Index: 953

Text #2

Max Range: 0.3215. Min Range: -0.3215

Max Act: 0.2734. Min Act: -0.0001

Data Index: 986859 (C4 (Web Text))

Max Activating Token Index: 763

Text #3

Max Range: 0.3215. Min Range: -0.3215

Max Act: 0.2809. Min Act: -0.0001

Data Index: 1285486 (C4 (Web Text))

Max Activating Token Index: 964

Text #4

Max Range: 0.3215. Min Range: -0.3215

Max Act: 0.2523. Min Act: -0.0001

Data Index: 195640 (C4 (Web Text))

Max Activating Token Index: 103

Text #5

Max Range: 0.3215. Min Range: -0.3215

Max Act: 0.2482. Min Act: -0.0001

Data Index: 398263 (C4 (Web Text))

Max Activating Token Index: 301

Text #6

Max Range: 0.3215. Min Range: -0.3215

Max Act: 0.2479. Min Act: -0.0001

Data Index: 649432 (C4 (Web Text))

Max Activating Token Index: 121

Text #7

Max Range: 0.3215. Min Range: -0.3215

Max Act: 0.2492. Min Act: -0.0001

Data Index: 389559 (C4 (Web Text))

Max Activating Token Index: 502

Text #8

Max Range: 0.3215. Min Range: -0.3215

Max Act: 0.2240. Min Act: -0.0001

Data Index: 1305020 (C4 (Web Text))

Max Activating Token Index: 412

Text #9

Max Range: 0.3215. Min Range: -0.3215

Max Act: 0.2268. Min Act: -0.0001

Data Index: 1002050 (C4 (Web Text))

Max Activating Token Index: 463

Text #10

Max Range: 0.3215. Min Range: -0.3215

Max Act: 0.2327. Min Act: -0.0001

Data Index: 629109 (C4 (Web Text))

Max Activating Token Index: 129

Text #11

Max Range: 0.3215. Min Range: -0.3215

Max Act: 0.2335. Min Act: -0.0000

Data Index: 1059605 (C4 (Web Text))

Max Activating Token Index: 453

Text #12

Max Range: 0.3215. Min Range: -0.3215

Max Act: 0.2271. Min Act: -0.0001

Data Index: 127890 (C4 (Web Text))

Max Activating Token Index: 982

Text #13

Max Range: 0.3215. Min Range: -0.3215

Max Act: 0.2140. Min Act: -0.0001

Data Index: 1118176 (C4 (Web Text))

Max Activating Token Index: 745

Text #14

Max Range: 0.3215. Min Range: -0.3215

Max Act: 0.2116. Min Act: -0.0001

Data Index: 1245088 (C4 (Web Text))

Max Activating Token Index: 468

Text #15

Max Range: 0.3215. Min Range: -0.3215

Max Act: 0.1907. Min Act: -0.0001

Data Index: 996961 (C4 (Web Text))

Max Activating Token Index: 850

Text #16

Max Range: 0.3215. Min Range: -0.3215

Max Act: 0.2034. Min Act: -0.0001

Data Index: 1305466 (C4 (Web Text))

Max Activating Token Index: 384

Text #17

Max Range: 0.3215. Min Range: -0.3215

Max Act: 0.2073. Min Act: -0.0001

Data Index: 245219 (C4 (Web Text))

Max Activating Token Index: 891

Text #18

Max Range: 0.3215. Min Range: -0.3215

Max Act: 0.2054. Min Act: -0.0001

Data Index: 367831 (C4 (Web Text))

Max Activating Token Index: 277

Text #19

Max Range: 0.3215. Min Range: -0.3215

Max Act: 0.1996. Min Act: -0.0001

Data Index: 1353457 (C4 (Web Text))

Max Activating Token Index: 841