Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2494 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.1804. Min Range: -0.1804

Max Act: 0.1693. Min Act: -0.0002

Data Index: 831283 (C4 (Web Text))

Max Activating Token Index: 238

Text #1

Max Range: 0.1804. Min Range: -0.1804

Max Act: 0.1804. Min Act: -0.0002

Data Index: 256952 (C4 (Web Text))

Max Activating Token Index: 189

Text #2

Max Range: 0.1804. Min Range: -0.1804

Max Act: 0.1540. Min Act: -0.0002

Data Index: 974977 (C4 (Web Text))

Max Activating Token Index: 722

Text #3

Max Range: 0.1804. Min Range: -0.1804

Max Act: 0.1403. Min Act: -0.0002

Data Index: 626856 (C4 (Web Text))

Max Activating Token Index: 113

Text #4

Max Range: 0.1804. Min Range: -0.1804

Max Act: 0.1590. Min Act: -0.0002

Data Index: 1010928 (C4 (Web Text))

Max Activating Token Index: 745

Text #5

Max Range: 0.1804. Min Range: -0.1804

Max Act: 0.1639. Min Act: -0.0002

Data Index: 536963 (C4 (Web Text))

Max Activating Token Index: 66

Text #6

Max Range: 0.1804. Min Range: -0.1804

Max Act: 0.1467. Min Act: -0.0002

Data Index: 318691 (C4 (Web Text))

Max Activating Token Index: 197

Text #7

Max Range: 0.1804. Min Range: -0.1804

Max Act: 0.1485. Min Act: -0.0002

Data Index: 228595 (C4 (Web Text))

Max Activating Token Index: 204

Text #8

Max Range: 0.1804. Min Range: -0.1804

Max Act: 0.1477. Min Act: -0.0002

Data Index: 18185 (C4 (Web Text))

Max Activating Token Index: 673

Text #9

Max Range: 0.1804. Min Range: -0.1804

Max Act: 0.1611. Min Act: -0.0002

Data Index: 508932 (C4 (Web Text))

Max Activating Token Index: 157

Text #10

Max Range: 0.1804. Min Range: -0.1804

Max Act: 0.1552. Min Act: -0.0002

Data Index: 1076909 (C4 (Web Text))

Max Activating Token Index: 121

Text #11

Max Range: 0.1804. Min Range: -0.1804

Max Act: 0.1545. Min Act: -0.0002

Data Index: 639962 (C4 (Web Text))

Max Activating Token Index: 242

Text #12

Max Range: 0.1804. Min Range: -0.1804

Max Act: 0.1594. Min Act: -0.0002

Data Index: 863620 (C4 (Web Text))

Max Activating Token Index: 713

Text #13

Max Range: 0.1804. Min Range: -0.1804

Max Act: 0.1579. Min Act: -0.0002

Data Index: 521754 (C4 (Web Text))

Max Activating Token Index: 296

Text #14

Max Range: 0.1804. Min Range: -0.1804

Max Act: 0.1420. Min Act: -0.0002

Data Index: 780642 (C4 (Web Text))

Max Activating Token Index: 95

Text #15

Max Range: 0.1804. Min Range: -0.1804

Max Act: 0.1397. Min Act: -0.0002

Data Index: 1313453 (C4 (Web Text))

Max Activating Token Index: 533

Text #16

Max Range: 0.1804. Min Range: -0.1804

Max Act: 0.1432. Min Act: -0.0002

Data Index: 1174738 (C4 (Web Text))

Max Activating Token Index: 500

Text #17

Max Range: 0.1804. Min Range: -0.1804

Max Act: 0.1469. Min Act: -0.0002

Data Index: 120745 (C4 (Web Text))

Max Activating Token Index: 399

Text #18

Max Range: 0.1804. Min Range: -0.1804

Max Act: 0.1504. Min Act: -0.0002

Data Index: 201804 (C4 (Web Text))

Max Activating Token Index: 133

Text #19

Max Range: 0.1804. Min Range: -0.1804

Max Act: 0.1353. Min Act: -0.0002

Data Index: 1273966 (C4 (Web Text))

Max Activating Token Index: 400